Blog on: Innovating your Evaluation Toolbox with World Café and Post-its
Many leading evaluators and authors have written about the important role of qualitative methods in evaluation (i.e. Michael Quinn Patton, Guba and Lincoln, Bessa Whitmore among many others).I would like to share with Evaltalk, a very innovative and efficient methodology I have introduced into my evaluation practice over the past two years that has consistently yielded spectacular results in terms of insight, energy, participation and buy-in. This methodology is not all encompassing but complements other methods used such as semi-structured interviewing, testimonials, field observation, quantitative data analysis and questionnaires to triangulate overall evaluation findings.
There are two parts to this exercise: 1) the first part fleshes out key results (by gender if you like); 2) the second part uses World Café to discuss key evaluation themes or issues of importance to the evaluation. Dot democracy is also used in both exercises to rank and prioritize results.The whole process takes no longer than 4 hours!
What are the benefits of these methods?
What is the role of the Evaluator?
When using this methodology, the evaluator essential becomes a facilitator; facilitating the group process and responding to questions. The evaluator should refrain from making any kind of judgements and remain totally neutral. Participants should be asked to analyze their own responses in plenary. This is not the job of the evaluator.
What is the World Café?
As defined by the World Café founders, “The World Café is an innovative yet simple methodology for hosting conversations about questions that matter. These conversations link and build on each other as people move between groups, cross-pollinate ideas, and discover new insights into the questions or issues that are most important in their life, work, or community. As a process, the World Café can evoke and make visible the collective intelligence of any group, thus increasing people's capacity for effective action in pursuit of common aims.(see: www.theworldcafe.com). This process can also be used in evaluations that are forward looking and may require an action plan for the future. This email describes world cafe used primarily in evaluation without the visioning session as this takes longer and is a different kettle of fish.
Who Participates?
A cross-section of stakeholders: decision-makers, technical staff, `beneficiaries`, programme officers, community leaders/members and/or other affiliated organizations that are very familiar with the programme or project being evaluated. Ideally up to a maximum of 24 people. Usually, beneficiaries form their own group so that their voices are not dominated or muted by more dominant personalities that can often take place with more mixed groups.
Where have these methods been used?
These methods have been used a dozen times in the last two years in three continents: Canada, Latin America and Africa in 6 countries with consistently energizing and spectacular results with participants with a high or low level of education and applied to institutional assessments, programme and project evaluations.
- Creates a very public forum for discussing what works, the challenges and recommendations for the future.
- Creates insight and energy.
- Demystifies the evaluation process; participants feel listened to and heard.
- Validates in a very public way what is happening in a given organization, programme or project.
- Helps reach overall group consensus of the most important trends and essentials in a project, programme or organization.
- Participants walk away with learning a new and innovative group process that they can use in their work.
What is the role of the Evaluator?
When using this methodology, the evaluator essential becomes a facilitator; facilitating the group process and responding to questions. The evaluator should refrain from making any kind of judgements and remain totally neutral. Participants should be asked to analyze their own responses in plenary. This is not the job of the evaluator.
What is the World Café?
As defined by the World Café founders, “The World Café is an innovative yet simple methodology for hosting conversations about questions that matter. These conversations link and build on each other as people move between groups, cross-pollinate ideas, and discover new insights into the questions or issues that are most important in their life, work, or community. As a process, the World Café can evoke and make visible the collective intelligence of any group, thus increasing people's capacity for effective action in pursuit of common aims.(see: www.theworldcafe.com). This process can also be used in evaluations that are forward looking and may require an action plan for the future. This email describes world cafe used primarily in evaluation without the visioning session as this takes longer and is a different kettle of fish.
Who Participates?
A cross-section of stakeholders: decision-makers, technical staff, `beneficiaries`, programme officers, community leaders/members and/or other affiliated organizations that are very familiar with the programme or project being evaluated. Ideally up to a maximum of 24 people. Usually, beneficiaries form their own group so that their voices are not dominated or muted by more dominant personalities that can often take place with more mixed groups.
Where have these methods been used?
These methods have been used a dozen times in the last two years in three continents: Canada, Latin America and Africa in 6 countries with consistently energizing and spectacular results with participants with a high or low level of education and applied to institutional assessments, programme and project evaluations.
Phase 1: Fleshing Out Key Results
The logic model is one important entry point to understanding what an organization hopes to achieve in terms of outputs, outcome and impact. But, what are the actual results at the mid-point or end of a programme and project? Do you have so many results that they are hard to prioritize? How do you know you are on the right track?
I have used post-its with groups to flesh out the key results of a programme or project. These can be used individually (3 post-its per person) or by table (5 post-its per small group of six). I begin by asking the participants, to reflect on their programme or project and to identify the 3 (for individuals) or 5 (per group) results that the programme or project has achieved.
For group post-its, I give the group 30 minutes to discuss the programme or project’s main achievements. They write their response on large post-its. For individual responses I give them 5 minutes to reflect and write the key achievements. During this time, the dots are distributed (1 green and two blue dots per person)
Plenary. When the group is finished, it is time for plenary. I ask the workshop participants who has a key result they would like to share with the group. They read out their answer and place it on the wall. Usually, there are others (either individually or a group) that have similar results. I ask those with similar results to also read out loud and to place the post-its underneath the previous ones on the wall. When a theme has been exhausted, I ask the group to name that theme and place it above on one of the empty post-its.
We then move on to another theme and I take a volunteer who would like to share their response. I ask if there are any others with the same or similar result. These are grouped together and then labelled by the group. This process continues until all themes have been exhausted. I then ask if there are any key results that might have been left out that people feel very strongly about. These are added if there are any.
The facilitator then summarizes the key themes up on the wall. The facilitator asks participants to now prioritize the most important results themes that are exhibited on the wall. They should first look at the themes and determine where they will place their dots. The green dot is the most important result theme.The two other green dots are place on their secondary and tertiary choices. Participants are invited to place their dots on the key result themes.
A plenary then takes place. The facilitator asks participants what they see from this exercise. What are the most important results? What are least important and why?
The evaluator can then use these findings as an important input into the key results of the programme or project which can then be triangulated and supplemented with the field observation, quantitative data and semi-structured interviews. The results can also be statistically analyzed using the dots in the form of pie charts or graphs.
Note to the facilitator:
The facilitator should refrain from giving too much information on the programme or project at the beginning of the workshop that might influence the key results. Remember that participants have been selected because they know the programme or project intimately.
You can disaggregate your results by gender by giving different coloured dots to men and women. Or dots could be distributed to distinguish management from programme staff and beneficiaries.
The logic model is one important entry point to understanding what an organization hopes to achieve in terms of outputs, outcome and impact. But, what are the actual results at the mid-point or end of a programme and project? Do you have so many results that they are hard to prioritize? How do you know you are on the right track?
I have used post-its with groups to flesh out the key results of a programme or project. These can be used individually (3 post-its per person) or by table (5 post-its per small group of six). I begin by asking the participants, to reflect on their programme or project and to identify the 3 (for individuals) or 5 (per group) results that the programme or project has achieved.
For group post-its, I give the group 30 minutes to discuss the programme or project’s main achievements. They write their response on large post-its. For individual responses I give them 5 minutes to reflect and write the key achievements. During this time, the dots are distributed (1 green and two blue dots per person)
Plenary. When the group is finished, it is time for plenary. I ask the workshop participants who has a key result they would like to share with the group. They read out their answer and place it on the wall. Usually, there are others (either individually or a group) that have similar results. I ask those with similar results to also read out loud and to place the post-its underneath the previous ones on the wall. When a theme has been exhausted, I ask the group to name that theme and place it above on one of the empty post-its.
We then move on to another theme and I take a volunteer who would like to share their response. I ask if there are any others with the same or similar result. These are grouped together and then labelled by the group. This process continues until all themes have been exhausted. I then ask if there are any key results that might have been left out that people feel very strongly about. These are added if there are any.
The facilitator then summarizes the key themes up on the wall. The facilitator asks participants to now prioritize the most important results themes that are exhibited on the wall. They should first look at the themes and determine where they will place their dots. The green dot is the most important result theme.The two other green dots are place on their secondary and tertiary choices. Participants are invited to place their dots on the key result themes.
A plenary then takes place. The facilitator asks participants what they see from this exercise. What are the most important results? What are least important and why?
The evaluator can then use these findings as an important input into the key results of the programme or project which can then be triangulated and supplemented with the field observation, quantitative data and semi-structured interviews. The results can also be statistically analyzed using the dots in the form of pie charts or graphs.
Note to the facilitator:
The facilitator should refrain from giving too much information on the programme or project at the beginning of the workshop that might influence the key results. Remember that participants have been selected because they know the programme or project intimately.
You can disaggregate your results by gender by giving different coloured dots to men and women. Or dots could be distributed to distinguish management from programme staff and beneficiaries.
Phase 2: Using the World Café Methodology in Evaluations
The World Café website as some excellent pictorials of the process that I use on flip-chart to explain the process. The methods are quite simple. Tables are arranged café style with up to six persons per table; a maximum of four tables. Each table has a theme drawn from the TORs of the evaluation: up to four key themes (15 -20 minutes for each rotation). These themes and questions are noted on the flip-chart at each table. Here are some examples that I have used.
Table Theme: Strengths and Weaknesses. Strengths and Weaknesses and Strategies for Overcoming Weaknesses or you could simply use a SWOT matrix: Strengths, Weaknesses, Opportunities and Threats
Table Theme: Lessons learned and Recommendations for the Future. What are the lessons learned of the programme to date and recommendations for the future.
Table Theme: Identify the Best Practices, Successful Achievements and/or Impacts with a Multiplier Effect. How has the programme contributed to the MDGs?
Table Theme: Key Activities to Achieve Project Results
-What core/essential activities of the project need to be consolidated in the next 1.5 years in order for the project to achieve its expected results?
-Are there any activities/results that should be suppressed or eliminated given time constraints, lack of resources, and/or a change in priorities?
Table Theme: Partnership. How could the partnership between Org X and Org Y be strengthened? Please give concrete suggestions!
Table theme: Sustainability. How can the project ensure that results and benefits will continue after the donor’s involvement ends?
Table Theme: Relevance. Is Organization’s mission still relevant? Are there elements of the mission needing updating?
Table Theme: Membership. 1. What do you perceive as Org. X role is for its members? 2. What are the key challenges facing Org’s membership in the next 5-10 years? 3. What will need to change for Org. X to meet these membership challenges?
Table Theme: Future Strategic Directions. What strategic directions do you see the organization taking in the next five years? What are the enablers and constraints to achieving this desired future?
If you have a questionnaire, I will distribute it for completion at the end of the four rotations before the plenary. Take advantage of the fact that you have a captive audience and a 100% response rate!
Getting Started
Explain the methodology to the group and what The World Café is all about. Explain that all 24 participants will have an opportunity to fully explore each theme. Every 20 minutes participants will be asked to rotate to a new table to explore a new theme and deepen the preceding groups work. Before a group rotates one person from that group will stay behind to explain all the responses to date noted on the flip-chart. Â After that person briefs the group of the responses to date (5 minutes), that person then rejoins his group that has rotated.
Explain to the group that each new group does not duplicate or repeat the previous group presentation, but deepens the previous group work, by noting new insights, different experiences or possibly different points of view on the flip-chart. These are noted usually with a different coloured marker.
See the note for the facilitator below about the distribution of dots.
Plenary
After all the rotations take place, it is time for plenary. The last tables to rotate will present all the responses to date. Explain to the participants that they will need to pay special attention to their colleagues presentation because at the end of plenary, they will need to use their dots to identify the most important points that have been raised during the World Cate. They will place one green dot on the most important point raised in ALL the flip-chart responses to date and up to three blue dots on other important points; but not as important as the Green dot.
One person from each table briefly reads ALL the flip-chart responses to date.The flip-chart can then be posted on the wall using masking tape. The second, third and fourth table presents their flip-chart responses and these are also posted on the wall. At the end of the presentations, you should have about 8 pieces of flip-chart paper on the wall (usually about 2 flip-chart paper responses per theme).
You now have a lot of information on the walls. How do you distinguish the most critical elements from less important ones? By letting the workshop participants vote! Remember, they know intimately the programme and project; what has worked and what hasn’t worked.
Ask participants to look at all the responses on the wall (3 minutes). They should begin to decide where they will be placing their dots. After a few minutes ask them to approach the wall and place their dots on the most important points. Make sure to get out of the way. You will note that participants will anxiously move to the wall to vote with their dots. After about 5 minutes, you will be ready for plenary.
Ask participants what they see on the wall. Start with the green dots by asking them what they see as the most essential important points highlighted. Participants will begin counting the dots themselves. Ask participants why they choose a given point. Make sure to note their comments. Move to the green dots and ask what they see and why they voted the way they did.
Explain to participants how these results will be used in the evaluation. I usually roll them up as part of the evaluation findings and explain this to them.
Thank participants for their participation.You can evaluate the 3.5-4 hour session using some other participatory evaluation tools (i.e. temperature gauge of the session or evaluation matrix) with the workshop participants.
In very little time, the evaluator/facilitator has created an open and public forum for discussing key results and evaluation themes critical to the evaluation that gives the evaluator much greater insight into the inner workings of a programme or project than would have been possible otherwise. Workshop participants feel energized from the process and feel that they have been heard and listened to, learned a new process that they can then use in their work and that their time and energy has been used very efficiently. A win-win for everyone.
Note to the facilitator:
The World Café website as some excellent pictorials of the process that I use on flip-chart to explain the process. The methods are quite simple. Tables are arranged café style with up to six persons per table; a maximum of four tables. Each table has a theme drawn from the TORs of the evaluation: up to four key themes (15 -20 minutes for each rotation). These themes and questions are noted on the flip-chart at each table. Here are some examples that I have used.
Table Theme: Strengths and Weaknesses. Strengths and Weaknesses and Strategies for Overcoming Weaknesses or you could simply use a SWOT matrix: Strengths, Weaknesses, Opportunities and Threats
Table Theme: Lessons learned and Recommendations for the Future. What are the lessons learned of the programme to date and recommendations for the future.
Table Theme: Identify the Best Practices, Successful Achievements and/or Impacts with a Multiplier Effect. How has the programme contributed to the MDGs?
Table Theme: Key Activities to Achieve Project Results
-What core/essential activities of the project need to be consolidated in the next 1.5 years in order for the project to achieve its expected results?
-Are there any activities/results that should be suppressed or eliminated given time constraints, lack of resources, and/or a change in priorities?
Table Theme: Partnership. How could the partnership between Org X and Org Y be strengthened? Please give concrete suggestions!
Table theme: Sustainability. How can the project ensure that results and benefits will continue after the donor’s involvement ends?
Table Theme: Relevance. Is Organization’s mission still relevant? Are there elements of the mission needing updating?
Table Theme: Membership. 1. What do you perceive as Org. X role is for its members? 2. What are the key challenges facing Org’s membership in the next 5-10 years? 3. What will need to change for Org. X to meet these membership challenges?
Table Theme: Future Strategic Directions. What strategic directions do you see the organization taking in the next five years? What are the enablers and constraints to achieving this desired future?
If you have a questionnaire, I will distribute it for completion at the end of the four rotations before the plenary. Take advantage of the fact that you have a captive audience and a 100% response rate!
Getting Started
Explain the methodology to the group and what The World Café is all about. Explain that all 24 participants will have an opportunity to fully explore each theme. Every 20 minutes participants will be asked to rotate to a new table to explore a new theme and deepen the preceding groups work. Before a group rotates one person from that group will stay behind to explain all the responses to date noted on the flip-chart. Â After that person briefs the group of the responses to date (5 minutes), that person then rejoins his group that has rotated.
Explain to the group that each new group does not duplicate or repeat the previous group presentation, but deepens the previous group work, by noting new insights, different experiences or possibly different points of view on the flip-chart. These are noted usually with a different coloured marker.
See the note for the facilitator below about the distribution of dots.
Plenary
After all the rotations take place, it is time for plenary. The last tables to rotate will present all the responses to date. Explain to the participants that they will need to pay special attention to their colleagues presentation because at the end of plenary, they will need to use their dots to identify the most important points that have been raised during the World Cate. They will place one green dot on the most important point raised in ALL the flip-chart responses to date and up to three blue dots on other important points; but not as important as the Green dot.
One person from each table briefly reads ALL the flip-chart responses to date.The flip-chart can then be posted on the wall using masking tape. The second, third and fourth table presents their flip-chart responses and these are also posted on the wall. At the end of the presentations, you should have about 8 pieces of flip-chart paper on the wall (usually about 2 flip-chart paper responses per theme).
You now have a lot of information on the walls. How do you distinguish the most critical elements from less important ones? By letting the workshop participants vote! Remember, they know intimately the programme and project; what has worked and what hasn’t worked.
Ask participants to look at all the responses on the wall (3 minutes). They should begin to decide where they will be placing their dots. After a few minutes ask them to approach the wall and place their dots on the most important points. Make sure to get out of the way. You will note that participants will anxiously move to the wall to vote with their dots. After about 5 minutes, you will be ready for plenary.
Ask participants what they see on the wall. Start with the green dots by asking them what they see as the most essential important points highlighted. Participants will begin counting the dots themselves. Ask participants why they choose a given point. Make sure to note their comments. Move to the green dots and ask what they see and why they voted the way they did.
Explain to participants how these results will be used in the evaluation. I usually roll them up as part of the evaluation findings and explain this to them.
Thank participants for their participation.You can evaluate the 3.5-4 hour session using some other participatory evaluation tools (i.e. temperature gauge of the session or evaluation matrix) with the workshop participants.
In very little time, the evaluator/facilitator has created an open and public forum for discussing key results and evaluation themes critical to the evaluation that gives the evaluator much greater insight into the inner workings of a programme or project than would have been possible otherwise. Workshop participants feel energized from the process and feel that they have been heard and listened to, learned a new process that they can then use in their work and that their time and energy has been used very efficiently. A win-win for everyone.
Note to the facilitator:
- After each rotation, change the colour of the markers at each table to distinguish each group.
- Dots are distributed to each person during the rotations. I green dot and up to three blue dots per person. Their use will be explained later at the beginning of plenary.
Advance Preparation:
Phase 1: Highlighting Key Results: 1.5 hours
Phase 2: Using World Cate: Â 2.5 hours
Facilitator’s notes:
It is better not to give individual exercise sheets. You want participants to talk and dialogue and not spend time filling in exercise sheets which they have a tendency to do. The focus should be on the discussion and flip-charting responses.
You may want someone to record and type up the results of each exercise which can then be distributed to the participants following the workshop.
Materials Needed and Logistics:
Phase 1: Highlighting Key Results: 1.5 hours
- Place on each table 5 large post-its and markers (or 3 large post-its per person).
- Place on an empty wall 5-6 blank large post-its which will become themes around the key groupings of results.
- Prepare your dots. 1 green and 2 blue dots per person to be distributed while participants are working on the results exercise.
Phase 2: Using World Cate: Â 2.5 hours
- Prepare Flip-charts in advance to save time. Questions and room to respond repeated on 2 pages. This is usually sufficient for 4 group rotations. Participants need not repeat answers of another group nor fill the page.
- Place cards with the evaluation theme on each table.
- Prepare dots in advance: 1 green dot and three blue dots per person.
Facilitator’s notes:
It is better not to give individual exercise sheets. You want participants to talk and dialogue and not spend time filling in exercise sheets which they have a tendency to do. The focus should be on the discussion and flip-charting responses.
You may want someone to record and type up the results of each exercise which can then be distributed to the participants following the workshop.
Materials Needed and Logistics:
- Large room with 4 round tables café style with 6 chairs each and a large wall space for posting flip-charts; table for facilitator’s materials.
- Large Post-its
- Scissors and coloured dots.
- A flip-chart for each table and variable markers.
- Large masking tape
- Cards for each table with each evaluation theme to be discussed.
- Large wall for the debriefing flip-chart papers and voting.
- For photos of the process, please check out the photo gallery at www.mosaic-net-intl.ca. For more information or assistance contact: Francoise Coupal at: fcoupal@mosaic-net-intl.ca.