Participatory Evaluation: A Case Study of CCIC Humanitarian Fund Projects in Haiti

by
Françoise Coupal and Marie Simoneau

Over the past decade there has been a growing body of literature in support of more qualitative methods and participatory approaches to evaluation. While articles and publications have mushroomed, there has been a paucity of case studies for development practitioners to draw and learn from. For development practitioners this has presented both challenges and limitations. Being faced with few "models" has forced us to be creative, to draw on experience and intuition of what does and does not work and finally, to learn to work with a degree of uncertainty.

The following case study seeks to present a Haitian experience of Participatory Evaluation undertaken in April of 1997. Challenged by the lack of models or experience the evaluation framework and its implementation was built from the ground up, with key project stakeholder involvement in all phases of the evaluation from planning, to training and implementation. It is hoped that this case study will help shed light on the unique approaches, resources, and skill sets that are required to undertake a participatory evaluation successfully.

Background

The Haitian Context

Haiti, sharing the western third of the island of Hispaniola with the Dominican Republic, is the poorest country in the Western Hemisphere. With a population of seven million inhabitants, 75 percent of the population live in abject poverty. Most Haitians live on tiny plots of land and survive through small-scale subsistence farming. The majority of the population does not have ready access to safe drinking water, adequate medical care, or sufficient food. The literacy rate is only 5 per cent. Life expectancy at birth is 53 years for men and 55 for women. The infant mortality rate is 105.7 deaths per 1000 births as compared to 6 deaths per 1000 births in Canada. Most of the country is mountainous and has been ravaged by extensive deforestation. Erosion is severe, accentuated by drought and unpredictable rain patterns.

The history of Haiti is one of sharply opposed interests and competing visions of state and nation. Since Haiti declared its independence from French control in 1804, it has been ruled by two powerful groups: military leaders and a merchant oligarchy. The vast majority of Haitians have been systematically excluded from meaningful participation in the development of their country. Their life experiences are largely determined by favoritism, corruption, arbitrariness, violence and repression.

In 1990, there were widespread expectations of political change. In the first democratic elections in the history of Haiti, Father Jean-Bertrand Aristide was elected President. His elections came on the heels of a popular mobilization that ultimately lead, in 1986, to the forced departure of Jean-Claude Duvalier. After only seven months in power, Aristide was overthrown in a bloody army coup on September 30, 1991. A wave of violence and terror swept the country over the next three years of military rule. An estimated 5000 people were murdered or simply disappeared; thousands fled the country or were forced into hiding. An international economic embargo caused enormous hardship and weakened the already fragile economic and social infrastructures.

With the U.S. invasion and return of President Aristide in October 1994, NGOs and grassroots organizations began to attempt to rebuild Haitian civil society and to implement development programs. However, economic and political problems remain: inflation, unemployment, malnutrition, drought, political instability, poverty, illiteracy, violence, insecurity, and weak democratic institutions all continue to undermine development efforts. A strict structural adjustment program may narrow the role of the state and set controls on government spending, privatize publicly owned enterprises, maintain low wages, and eliminate import tariffs.

The Canada-Haiti Humanitarian Alliance Fund

It was within this particularly troubled political, economic and social conjuncture that, in 1973 the Canada-Haiti Humanitarian Alliance Fund was developed by the Canadian Council for International Cooperation (CCIC). This coalition of volunteer Canadian organizations is committed to world development through projects promoting social and economic equity, democratic participation, environmental integrity, and respect for human rights. Financed by the Canadian International Development Agency (CIDA) with a budget of CDN $ 4.5 million over four and half year period, the fund's objective was to allow Canadian NGOs with programs and partners in Haiti to continue their development and post-disaster activities in spite of the 1991 coup d'état and the effects of an international embargo. The fund was seen as a mechanism for supporting the work of local NGOs within civil society during a period of political instability and severe repression.

Thirty-six projects were funded in various sectors, including agriculture and reforestation, health and nutrition, sanitation, credit, institutional strengthening, street children, and communications. Fifteen projects were approved during the period of the coup and the twenty-one remaining were implemented immediately after Aristide's return.

Why a Participatory Evaluation?

Customarily, evaluations serve several functions. Traditionally, the central function has been establish accountability to ensure that funds have been spent wisely and in accordance with their stated purpose. This function depends on an underlying assumption: that the people passing judgement on a project, program, or organization need to be a neutral and independent third party. Only then can an evaluation be considered "objective" and "scientific". Over time, however, this function has come to be viewed as too narrow and inadequate given the time and costs involved in development projects. The extractive process of traditional evaluations has been a source of complaint and is now thought to have little enduring and positive impact for projects.

The diagram below demonstrates that participatory evaluations are more multidimensional than traditional ones. In this approach, accountability is one of many functions and should also serve project beneficiaries.

Adapted from Who are the Question-makers?, UNDP, 1997.

In the case of the Canada-Haiti Humanitarian Alliance Fund, initial interest in undertaking a participatory evaluation came from CCIC whose staff were familiar with qualitative methods and approaches to evaluation. Yet they wanted to go beyond the accountability function to build capacity among Haitian and Canadian NGOs by training local participants in participatory methodology and approaches. The evaluation itself was presented as a learning process and a way of examining projects to see how they could be made more effective. The evaluation would also allow CCIC to assess the efficiency and the actual impacts of the fund. For example, did the fund really enable Canadian and Haitian NGOs to continue functioning during the coup? Did funded projects improve the well-being of the beneficiaries? Were Haitian civil society and development NGOs made stronger by these projects?

The main challenges faced in planning the participatory evaluation were:

• the integration of new methods within the limited time and resources accorded to the evaluation
• the coordination of more than 36 projects, 40 Canadian and Haitian partners dispersed over 4 disparate regions
• a lack of linkages among these projects
• the broad range of activities: from preventive health programs, credit, communications and agriculture, to reforestation
• the variance in cost of the projects: from less than CDN $ 5,000 to more than $ 250,000
• availability and capacity of NGOs to participate in a three-week intense training and evaluation
• the choice of appropriate program staff for training in participatory evaluation methods
• the logistical challenges given the dispersion of projects
• timelines of the projects - some had been completed previously and others were still in progress
• cultural and language barriers: Creole is first language of Haitians, making the translation of materials into both French and Creole a necessity
• the possibility of creating a situation where program staff feel threatened by the evaluation process
• the lack of Haitian co-facilitators familiar with Participatory Rural Appraisal (PRA) methods and tools.

Given these challenges, the preparatory and training phases of the evaluation would be crucial to its eventual success.

While the initial impetus for review came from the funding organization, the final decision as to the type and scope of evaluation rested with the project stakeholders attending the planning meeting. They decided on the type of evaluation they wanted.

The Overall Framework for the Participatory Evaluation

From the outset, the parameters for evaluation were defined with over 30 project stakeholders (Directors and/or program staff of NGOs funded by the Program). Also, there was a great deal of latitude given by the funding agency to develop the framework iteratively, based on feedback from the Haitian partners. Two missions were undertaken to Haiti. The first sought to plan and "flesh out" the framework of the participatory evaluation, while the second focused more on training and the actual launching of the evaluation. One can distinguish at least four key process phases: the planning mission, training, visiting project sites, and collective reflection and dissemination of the findings.

Salient Features of the Participatory Evaluation

• 15 projects were evaluated over a three -week period with teams of three to five persons per project;
• The selection of projects and the themes to be examined were determined in a planning meeting with project stakeholders;
• 28 participants were drawn from NGOs directly linked to the projects and trained to become PE facilitators;
• Efforts to achieve gender balance among the participants was promoted from the outset;
• An intense training over four days with daily practice within the community was essential to the learning process. The residential training took place outside of the capital Port-au-Prince in a rural setting;
• The evaluators were project stakeholders involved in the collective reflection and discussion of the project's achievements;
• Participants or PE facilitators drawn from the projects were not directly involved in the evaluation of their own projects;
• When selecting participants for the training in PE methods, a person's openness and willingness to learn new methods was deemed more important than a given level of education or prior experience.

I: The Planning Mission

A one-week mission took place in February 1997 to plan for the evaluation. A first meeting was held with 36 NGOs and several community representatives participating. This meeting initiated participants into the world of participatory evaluation by using participatory exercises throughout the day. The forum created an atmosphere of enthusiasm and intrigue as the Haitian NGOs were accustomed to more traditional meetings and presentations.

The planning meeting sought to answer some key questions:

• Was there a commitment to undertake a participatory evaluation?
• If so, what resources and support could be expected from the local NGOs?
• Who in terms of profile and skills should be involved in the evaluation?
• Why were we undertaking an evaluation and were its objectives?
• What indicators should be used?
• When should the evaluation take place?
• Where should the training take place?

An important exercise determined which type of evaluation was of greatest interest to participants. Small groups were asked to discuss and draw their experiences of evaluation. That this exercise proved very revealing can be seen from the following pictures:

The first picture is entitled Who Decides? An eye (the donor) looks from afar. A scale is shown holding people on one side and dollars on the other. The scale tips toward money that also symbolizes power. While people are everywhere, "who decides" remains the critical question.

In the second picture, a well -dressed evaluator sits behind a desk with file drawers full of reports. The project stakeholders are outside, in the community. Arrows indicate the flow of information. The "expert" may write a report for the donor but the project stakeholders are clearly the objects of evaluation.

This depiction of a participatory evaluation shows, in one corner, a meeting of all the project stakeholders. Following the meeting, the group goes into the community to meet project beneficiaries, individually or in small groups, under a tree or at a workplace. As one of the authors of the drawing explained, in a participatory evaluation "there is an exchange of ideas, experiences and a group discussion of the problems and solutions. The results of the evaluation are shared with all of the key stakeholders."

This diagram was drawn at the end of the participatory evaluation process. It represents the four key stakeholders: the facilitator, the intermediary (NGO), the project beneficiaries and the donor. As noted by one of the authors of the drawing: "an implemented project is the convergence of different stakeholder visions." Here, a sense of equity, balance and convergence is seen as part of the participatory evaluation process.

These drawings helped to focus discussion on the type of evaluation NGOs preferred: the more traditional evaluation where a team of "experts" are sent in, or a more participatory process involving project stakeholders in a process of collective reflection and analysis.

The selection of projects was done through the simple method of pulling names out of a hat. A number of NGOs also asked that their projects be included in the evaluation process, and these were added. The names of the projects were then entered onto a matrix to ensure a sample that would be representative in terms of the size of project, its geographic region, and sectoral focus.

Participants were then asked to define three issues that they would like the evaluation to examine. Each response was placed on a colored post-it note and regrouped thematically by the participants themselves. These preliminary themes guided the evaluation exercise and included some of the following issues: the impact of the project, its sustainability and viability, gender equity, and the project's relationship with the community and other NGO partners.

Also discussed at the meeting were the timeline for evaluation and the level of commitment one could expect from different stakeholders. Commitment took various forms: the provision of transportation and accommodation during the evaluation, or actual participation in the actual training and evaluation phases. The purpose of this approach and these exercises was to demystify the whole process of evaluation and ensure an early buy-in and interested commitment to it.

The results of this planning meeting were shared with all the Canadian and Haitian NGOs and they were encouraged to contribute. Those who could not take part in the training phase were encouraged to participate in the final "restitution," a half-day meeting where the results of project evaluations and an assessment by participants would be exchanged.

Phase II: Training the Participants

Twenty-eight participants were trained in Participatory Evaluation by two facilitators using Participatory Rural Appraisal (PRA), also known as Participatory, Learning and Action (PLA) approaches. Both PRA and PLA were pioneered in the South and promoted by Robert Chambers, at the Institute of Development Studies (IDS), and the International Institute for Environment and Development (IIED) both in England. PRA and PLA cover a family of approaches that seek "the full participation of people in the processes of learning about their needs and opportunities, and in the action required to address them." Participatory approaches are rooted by a belief in the innate wisdom of people, particularly poor people. PRA and PLA use visual tools and local materials (rocks, stones, seeds, beans) to generate discussion and analysis among participants.

An Overview of Training

The four-day workshop was entirely experiential, and exercises were designed to stimulate learning around participatory evaluation and the key concepts and tools of participation. Exercises involved skits, role-playing, sorting exercises, drawings, small group discussions and hands-on exercises such as matrices, ranking, community mapping, force-field analysis, historical timelines, gender division of labor, transect walks, etc. Some examples follow:

Participatory vs. Traditional Evaluations.

To help participants understand the difference between participatory and traditional evaluations, short descriptions of each approach were noted on individual strips of paper. Thirty strips of paper were distributed to each group, who read such short statements such as: "donor control of resources and decisions," "control and decisions by program/project beneficiary," "collective learning process," "an extractive process, outsider's perspective," "stakeholders are the question-makers," etc. These statements were then discussed within the group and sorted into two categories: participatory or traditional evaluation. Similar exercises were done using open and closed questions followed by mock interviews.

Role-playing

Role-playing was used in a number of exercises to make participants aware of how certain attitudes and behaviors can have an adverse effect on others. Groups of three participants acted as dominators, the dominated, as listeners and even as saboteurs. Discussion followed as to how it felt to be dominated or sabotaged, and strategies for neutralizing dominators and saboteurs were addressed. These exercises are excellent energizers, and make participants aware of social dynamics and class differences. Those who dominate often have a higher level of education, social standing or prestige and tend to prevent poorer participants from for speaking for themselves. It is essential for a facilitator to spot dominators and saboteurs early on, and to ensure that evaluation findings are not skewed in their favor.

Mapping

This exercise involves asking community members to "map" their community from a local perspective. Often what goes unmentioned by participants can be as important as what they note on this map. Community members may also highlight key areas of project activity or distinguish different socio-economic groupings. Mapping is an excellent tool to use when first arriving in a community and can provide an important insiders perspective.

Triangulation

Triangulation is essential for ensuring that evaluation findings are valid, reliable, and representative. Participatory evaluations seek to triangulate findings using different tools and methods, ensuring that a wide cross section of project stakeholders and beneficiaries are involved in the process.

The cumulative effect of these exercises gave participants the experience to become participatory evaluation (PE) facilitators. Combined with daily excursions into the community, the exercises allowed participants to apply what they had learned in the field. If participants were having difficulty with PE concepts, lead facilitators of the workshop discreetly worked with them during the exercises.

Another important exercise was to explore collectively the attitudes and personal qualities of facilitators, since they play a critical role with the community in creating a climate of trust. The ideal qualities of a facilitator include: an open mind, being a good listener, allowing others to speak, patience, trust in people's abilities, and valuing the opinions of others. Role-playing and practical exercises in the community setting enabled participants to develop their facilitation skills.

"Handing Over the Stick" and Overcoming Resistance

"Handing over the stick" has tremendous symbolic value in participatory methods. Often, PRA uses a stick or a marker to draw visual representations. The question of who holds the stick is quite significant, as it is representationally akin to holding power.

Early on in the training, a key obstacles was the belief of PE facilitators that poor or illiterate people could not map their community or make matrices for themselves. Participants tended to guard their stick or marker, and they felt they needed to act as interpreters rather than letting the people use the tools themselves. Resistance to handing over the stick was particularly strong during the first few days of training but diminished over time, as facilitators gained confidence and realized, to their surprise, that poor people could in fact participate quite effectively. Throughout the training phase, facilitators were urged to keep an open mind and to continually practice the tools. Individuals in the community could not be expected to use the tools if facilitators did not hand over their stick. Trainers also stressed the need for participants to work on their own internal blockages and areas of resistance.

Phase III: Visits to Projects

A total of 15 projects were visited by the PE facilitators over a two-week period. Each facilitator visited a total of two projects, and spent three days living on the project site. The group of 29 facilitators was divided into teams of three to five persons. PE facilitators articulated their three top priorities, and each team was composed to ensure gender balance, openness, facility with the methods, and sectoral expertise in the project area.

The first eight projects visited were in the northern part of the country, principally around the region of St. Michel de l'Attalaye. Each team had a number of evaluation questions or issues to explore with the community, such as the impact of the project, its relationship with other partners (CCIC, NGOs, the community), sustainability and viability, efficiency, project management, the role of women, and gender equity. A number of sub-questions were defined by the participants for each of the overall themes.

Phase IV: Collective Reflection and Dissemination

Each team was responsible for consolidating their community work into a brief project report. Each report sought to summarize their key findings as well as to integrate the drawings of the community. Where drawings were made on the ground, the team would replicate the drawing on paper. At the end of their visit, some teams debriefed those project stakeholders present in the community to cross-check the reliability of their findings. In turn, each team was responsible for making a presentation of their findings to the larger group. These presentations were an important source of mutual education, given the active exchange of learning experiences.

All the program partners were invited to attend a final presentation organized by the PE facilitators. It involved showcasing the drawings, maps and findings of each project evaluated, role-playing, and providing testimonials of their experiences.

Key Outcomes and Findings

The participatory evaluation generated a number of significant findings and lessons at various levels:

1. Excellent logistical and administrative support is essential;
2. Allow sufficient down time at the end of project visits for group reflection and lessons learned (2 days);
3. Ensuring a good gender mix is critical to the evaluation process;
4. Some learners take longer than others and special assistance may be required to help them over their own internal and cultural blockages;
5. Training outside of the capital city in modest surroundings can contribute significantly to developing a team atmosphere and learning to adapt to different living conditions;
6. Seek to "walk the talk" and promote cultural sensitivity and good participatory practices (eating local food, sleeping on mats, handing over the stick and listening to others);
7. Don't be rushed. Some of the best interviews are those where one takes time to build trust and rapport are achieved.

Building Capacity

One of the key outcomes of the participatory evaluation was the building of capacity. At a personal level, the group of 29 trainees was transformed. Their outlook on poor people had changed significantly. The facilitators had become much more aware of the constraints that poor people face, and of the ways that they are subordinated. Their understanding and appreciation of qualitative methods, and of the importance of living in the community were also sharpened. This personal transformation was best expressed by the participants themselves. When asked to comment on the training they had received and their discoveries, they offered some of the following comments (Coupal, 1997, pp. 11-12):

• "The training allowed me to eliminate my intellectual superiority vis-à-vis illiterate people."
• "I discovered other methods to help stakeholders evaluate their projects."
• "The training allowed me to discover tools to use to overcome obstacles between staff and project beneficiaries."
• "The importance of the tools; the importance of working in a team; the contribution of the NGO in the community; and how to facilitate a group."
• "I was able to discover with these tools that everyone has the potential and it is at that moment that one realizes the capacity and potential of community members."
• "Participatory evaluations go further than traditional evaluations. Participatory evaluations create better participation in projects."
• "The evaluation enabled me to learn more about my colleagues and the capacity and cultural vibrancy of the community."
• "The training made us dynamic."
• "I reached my initial objectives; I recognized my limits; I must continue to practice participatory methods."
• "The training enabled me to establish a demarcation line between participatory and traditional evaluations."
• "From the cultural perspective, the community taught me many things."
• "Illiterate people can let images speak for themselves as much as literate people."

While training laid the foundation for participatory evaluation, the real growth took place during daily excursions into the community and during visits to the projects. That is where the PE facilitators realized the importance of gaining trust and establishing rapport before introducing tools or asking questions. They also learned that members of a given community and project were quite capable of using matrices and making maps. However, their impact on the community was more difficult to gauge. Certainly there was a great deal of curiosity and intrigue during these team visits to the projects. Some project beneficiaries felt a strong sense of ownership for their drawings, and adorned the walls of their offices with them.

The Learning Curve

The timeline that follows maps out the major breakthroughs in the training and participatory evaluation process. It is interesting to note that the facilitators' perceptions of poor people had been transformed dramatically by the end of the participatory evaluation. The facilitators began the training saying that poor people could not use the stick or pick up a pen. By the end of the training phase, the facilitators felt that "rural people were more creative and spontaneous than urban people," and that the "real danger to development were literate people without awareness."

Beginning of Training:

Day 1:

Overall enthusiasm and interest in participatory methods, however there remains an underlying skepticism that these methods and tools can actually be applied to the Haitian context. There is the feeling among the participants that poor people can not hold the stick or draw for themselves. The problem is one on the outside- seen as one belonging to illiterate people vs. themselves.

Day 2:

Day long session on tools done on sand using local materials. The practical side of the exercises help the participants get away from paper and pen.

Day 3:

One begins to see an assimilation of the tools and concepts. At least half the group is successful in letting community persons use the tools for the community.

The world of illiterate people is being discovered. There is acknowledgment that the problem may not be outsiders, but themselves given their own personal blockages. The team begins to solidify.

Departure for the Projects:

Day 4:

Departure for project sites. General openness to experiment with the tools.

Day 7:

Return from first project visit. Overall enthusiasm and discovery that the tools work! The PE facilitators begin to create their own terminology for tools such as a zing sag walk to replace transect walk to take into account all the twists and turns in a promenade.

The PE facilitators still require prodding in terms of keeping their focus on the evaluation themes vs. the tools. There is a tendency to use the tools mechanically.

A PE facilitator remarks that "illiterate people are not stupid" while another remarks that the "real danger to development are literate people who are not aware."

Day 16:

After visiting their second project, participants begin to compare the use of tools by rural and urban people. They conclude that rural people are more spontaneous and creative.

There are overall improvements in project reports (i.e. better integration of tools with evaluation findings) and a personal transformation in the PE facilitators. Many have gained self-confidence. There is a commitment to form a participatory network in Haiti.

Data Interpretation and Presentation Stage

Data were interpreted at three points: (1) after the first series of project visits; (2) after the second series of project visits; and (3) during a final presentation open to all Fund partners. Each team was also responsible for communicating the results to the community they had visited before leaving, although this was not done in every case.

With respect to interpretation of the data, we observed an improvement in the presentations. Increasing emphasis was placed on the evaluation themes rather than on the tools used. There was also a greater mastery of the general approach, of open-ended questions and of the ways to address sometimes delicate themes.

The facilitators learned a number of lessons from their first practical exercises in the community:

1. it is essential to use the tools;
2. one must adapt the tools to conversations, vs. simply practicing the tools;
3. individuals' trust should be earned before presenting the tools;
4. it is important to get people's attention;
5. in a rural area, physical contact or position can be more important than direct looks because peasants do not tend to look each other in the eye;
6. the availability of interviewees of both genders is also important. It is harder to use PRA methods when women are very busy;
7. the initial strategy for approaching community members is very important.

The Program's Findings

The projects being evaluated covered a wide range of sectors, from microcredit for women, to agriculture, health, and social support for street children. All of the projects involved different NGOs from both Haiti and Canada, and thus their aims and impacts diverged greatly. Among the projects evaluated, it was estimated that over 10,000 persons benefited from the program. Some projects provided support to grass-roots groups of 200 people or more, which allowed the Fund to expand its coverage quite significantly. Since the program was established at a critical juncture in Haiti's political development, it also created opportunities for local groups to come together and organize.

Near the end of the evaluation process the PE facilitators were asked to highlight, based on their recent experience, those factors that contribute to or hamper the realization of project objectives. According to the PE facilitators, a number of factors contributed to a project's success (Coupal, 1997: 23):

• it must respond to real needs in the community;
• the training provided by the program should help to create and build capacity of organizations and groups;
• group dynamics should enhance the feelings of belonging and honesty;
• community participation from the onset is important;
• good governance and leadership of the project is key;
• good project structure and follow-up is essential;
• objectives should be clear and well defined;
• strategies must be flexible;
• one must have sufficient resources.

Among the obstacles that prevent or limit the achievement of project aims, the following was noted:

• the centralization of power, or an inadequate sharing of information about the project;
• poor management and lack of transparency;
• poor leadership and paternalistic attitudes;
• egotism;
• a lack of community participation and communication between project beneficiaries and project managers;
• poor monitoring and follow-up.

The PE facilitators felt strongly that successful projects were those that integrated participation throughout the project cycle, and that projects without participation had a limited impact on the community. The evaluation also noted the importance of networking and cross-fertilization when stakeholders are brought together over issues of common interest and concern.

Limitations and Difficulties

In order to be really effective, participatory approaches must be both inherent in the NGO's programming and an organic component of project design. Many of the projects evaluated in this study were neither designed nor implemented in a participatory way. We hope that these participatory evaluation results will be used to refine key issues and generate new questions to be addressed in the next phase of the projects.

One crucial limitation of this evaluation was the absence of some key stakeholders. Even though project partners were informed in advance of the evaluation and asked to facilitate the work of the PE teams, some of these stakeholders were unable or unwilling to comply. Certainly more than one field visit needs to be made, and more coordination before visits is essential in order to guarantee the involvement of all stakeholders. Additional time should be allocated to field visits and to the consolidation of findings stages of any evaluation.

In preparing for the evaluation, it was discovered that local expertise in PRA/PLA was next to non-existent in Haiti. In addition, the quality of the teams of PE facilitators varied enormously. This discrepancy was reflected in the unevenness of the project evaluation reports, especially in the first phase when participants were learning and applying the tools.

Conclusion

The Haitian experience revealed that participatory evaluations are nevertheless a viable alternative to more traditional approaches to evaluation. Over a one-month period 29 participants drawn from the projects and their respective communities were trained in PE methods and approaches. During this time, 15 projects were evaluated in various sectors including agriculture, credit, health, children and communications.

A number of arguments are often heard to discourage the use of participatory evaluation:

1. participatory evaluations can only be done with projects designed in a participatory manner;
2. participation takes time and is costly;
3. project participants cannot "objectively" evaluate projects;
4. evaluations require "experts."

However, this Haitian case of participatory evaluation debunked a number of objections. The CCIC Fund was traditionally designed so as to respond to donor priorities and initiatives. While there was some stakeholder consultation, the design and implementation of individual projects varied enormously, and some projects were not explicitly participatory. Introducing a participatory approach to evaluation did not hinder or disrupt the status quo. On the contrary, the PE facilitators were able to better appreciate the importance of participation for all aspects of a project's implementation. As well, project stakeholders became quite interested in the methods and tools used.

While the cost of this evaluation (around US $ 65,000) is comparable to more traditional evaluations, its impact was significantly greater than a traditional evaluation, both at a personal and a professional level. The PE really changed the way that NGO PE facilitators viewed development. The three weeks required to undertake this evaluation is comparable to traditional evaluations, and possibly shorter, given the number and complexity of projects. One advantage in this PE was the large number of facilitators, which enabled us to cover a large number of projects in a short period of time.

While many of the PE facilitators had thus been the objects of evaluation, most were not (at least initially) familiar with participatory approaches and methods. However, this participatory evaluation did show that evaluations do not require outside "experts"; a core team of experienced and skilled participatory practitioners can effectively train others to use and apply participatory methods to participatory evaluations. While the quality of individual reports did vary, for the most part the teams captured what was happening in the projects.

The participants in this evaluation have now formed their own network so as to continue using and applying participatory tools and methods, and to exchange experiences. They have met on a regular basis since the evaluation. They are also interested in offering their learned expertise to other organizations who may require a consultant.

Many have integrated PE tools into their day-to-day work with organizations and communities. Others use these tools with their families and children. One participant noted: "I am much more conscious of open vs. leading questions. So much so, that when my wife asks a question I reformulate it so that it is an open question."

Certainly all the participants had an experience that transformed their perception of poor people, from one of helpless individuals to one of creative, spontaneous and "included" individuals. There is also a wider recognition of the importance of participation throughout the project cycle, from project design to implementation to monitoring.

Another participant noted that facilitators now have a better understanding of what is meant by sustainability and viability. This has helped them in discussions with donors.

This participatory evaluation created a unique bond between the participatory facilitators. They now act to support and encourage one another. As many of these facilitators work for different NGOs, they have also become important points for contact and exchange.

Finally, there are a number of questions and challenges that lie ahead. Will the newly formed participatory network enable these new practitioners to continue to learn through action and debate, to hone their skills and deepen understanding? Have their behaviors and attitudes changed in a meaningful and lasting way? Will the tools be used to empower local people and give them a say in projects affecting them, and can they make decisions that will influence other stakeholders? Will they be able to address deeper underlying issues in their communities? If even some of these objectives are attained, this evaluation will have succeeded in building PRA/PLA capacity among NGOs in Haiti, and in empowering the poor and weak.

If you have any questions about this article or participatory evaluation, please feel free to contact coupal@mosaic-net-intl.ca or marie-s@wusc.ca.

Bibliography

Anayanwn, C.N. The Technique of Participatory Research in Community Development. Community Development Journal, January 1988, Vol. 23: 11-15.

Armstrong, John; Key, Michael. Evaluation, Change and Community Work. Community Development Journal, October 1979, Vol. 14:210-223.

Campos, Juanita Diane. Towards Participatory Evaluation An Inquiry into Post-Training Experiences of Guatemalan Community Development Workers. Dissertation submitted to University of Massachusetts, School of Education, May 1990.

Chambers, Robert. (1994). "Participatory Rural Appraisal (PRA): Analysis of Experience", in World Development, Vol. 22, No.9, pp.1253-1268, 1994.

_______________. "Shortcut Methods for Gathering Social Information for Rural Development Projects", in M. Cernea. Putting People First, Oxford University Press, Oxford.

Clark, Noreen; McCaffery, James. Demystifying Evaluation. World Education, Boston, Massachusetts, 1979.

Coupal, Françoise. Participatory Evaluation of the Canadian Haitian Humanitarian Alliance Fund, Final Report, July 1997. Haiti. Report submitted to the Canadian Council for International Cooperation, Ottawa, Canada. July 1997.

________________. "Participatory Project Design: Its Implications for Evaluation. A Case Study from El Salvador." Paper presented at the 1995 Evaluation Conference, Vancouver.

Cuthbert, Marlene. Evaluation Encounters in Third World Settings. In M.Q. Patton (Ed.), Culture and Evaluation. San Francisco, Jossey-Boss, pp. 29-35.

DAC Expert Group on Aid Evaluation. Evaluation and Participation. An Approach Paper. Development Studies Unit, Department of Social Anthropology, Stockholm University, September 1995.

Davis-Case, D'Arcy. The Community's Tool Box. The Idea, Methods and Tools for Participatory Assessment, Monitoring and Evaluation in Community Forestry. Community Forestry Field Manual 2, FAO, Rome, 1990.

________________. Community Forestry. Participatory Assessment, Monitoring and Evaluation. Community Forestry Note 2, FAO, Rome, 1989.

FAO. Participatory Monitoring and Evaluation. Handbook for Training Field Workers. FAO, Regional Office for Asia and the Pacific, FAO, Bangkok, Rapa Publication: 1988/2.

Derricourt, Nick; Oakley, Peter (Eds.) "Participation in Evaluation", Special Issue, Community Development Journal, Vol.23:1, 1988.

FAO. Community Forestry--Participatory Assessment, Monitoring and Evaluation. FAO, Rome, 1989.

Fernandes, Walter; Tandon, Rajesh. Participatory Research and Evaluation-Experiments in Research as a Process of Liberation. Indian Social Institute, New Delhi, Indian Social Institute, 1981.

Fetterman, David. (Eds.). Qualitative Approaches to Evaluation in Education. New York, Praeger Press, 1988.

Feuerstein, Marie-Thérèse. "Finding the Methods to Fit the People: Training for Participatory Evaluation", Community Development Journal, 1988, Vol. 23: 16-25.

_______________________. Partners in Evaluation, Evaluating Development and Community Programmes with Participants. McMillan, London, 1986.

_______________________. Participatory Evaluation: by, with and for the People. Idea: Reading Rural Development Communications Bulletin. London: University of Reading Agricultural and Extension and Rural Development Centre, April 1982, Vol. 14: 18-23.

_______________________. "Community Participation in Evaluation: Problems and Potentials, International Nursing Review, 1980, Vol 27 (6), 187-190.

_______________________. "Evaluation by the People." International Nursing Review, Vol. 25 (5), 1978, 146-153.

Forss, Kim; Rebien, Claus. "Evaluation with (and without) Participation. A Case Study of the Evaluation of the UNDP project URT/89/003-Strengthening the Ministry of Water-Towards Achieving IWSSD Objectives." Report, New York, UNDP, 1992.

Forss, Kim. Report from the Participatory Evaluation of the Royal Government of Eland/UNDP Project ELA/90/003-Rural Water Supply and Sanitation. September 6, 1994, UNDP.

_____________. Participatory Evaluation: Questions and Issues. A Report Commissioned by the Central Evaluation Office, Occasional Paper No.1, UNDP, New York, October 1989.

Freedman, Jim. Participatory Evaluations-Making Projects Work. Technical Paper No. TP94/2, Division of International Development, International Centre, The University of Calgary, Canada, 1994.

Gariba, Sulley. "Participatory Impact Assessment as a Tool for Change: Lessons from Recent Experience in Poverty Alleviation Projects in Africa." Paper presented at 1995 Evaluation Conference, November 1-5, 1995, Vancouver.

Ghai, D.; Westendorff, D.(Eds.). Monitoring Social Progress in the 1990s: Data Constraints, Concerns and Priorities. UNRISD, Avebury, Aldershot, 1994.

Guba, Egon; Lincoln, Yvonna. Fourth Generation Evaluation. Age Publications, England, 1990.

________________________. "Do Inquiry Paradigms Imply Inquiry methodologies?" in David Fetterman (Ed.). Qualitative Approaches to Evaluation in Education: The Silent Revolution. New York, Praeger Press, 1988, pp.89-115.

Gupta, V.K. Monitoring and Evaluation of Small Farmers' Cooperatives in Asia. FAO, 1989.

Hall, Budd; Etherington, Alan; Jackson, Ted. "Evaluation, Participation and Community Health Care: Critique and Lessons." Paper presented at the American Health Association, Toronto, Canada, November 1979.

Jackson, Edward. "Participatory Impact Assessment for Poverty Alleviation: Opportunities for Communities and Development Agencies." Paper presented at 1995 Evaluation Conference, November 1-5, 1995, Vancouver.

Jackson, Edward; Kassam, Yusuf. Better Knowledge, Better Results: Participatory Evaluation in Development Cooperation. 1996.

Kassam, Yusuf. "The Combined Use of Participatory and Survey Methodology in Evaluating Development Projects. A Case Study of a Rural Development Project in Bangladesh." September 1995. Paper presented at 1995 Evaluation Conference, November 1-5, 1995, Vancouver.

Kensey, David. Evaluation in Nonformal Education. The Need for Practitioner Evaluation. Amherst, Mass., The Centre for International Education, University of Massachusetts, 1981.

_____________. "Participatory Evaluation in Adult and Nonformal Education." Adult Education. Vol. 31: 155-168.

Jackson, E. T.; Yussuf, Kassam. Better Knowledge, Better Results: Participatory Evaluation in Development Cooperation, 1996.

Krishnamurthy, Lakshmi. "Exploding Myths: The Participants Evaluate." In W. Fernandes and R. Tandon, (Eds.). Participatory Research and Evaluation: Experiments in Research as a Process of Liberation. New Delhi, Indian Social Institute, pp. 151-165.

Krueger, R.A. Focus Groups: A Practical Guide for Applied Research. Sage, Newbury Park, 1988.

Lackey, Alvin; Peterson, Mack; Pine, Jeff. "Participatory Evaluation: A Tool for Community Development Practitioners." Journal of the Community Development Society. 12pp, 1981.]

Merrifield, Juliet. "Putting the Scientists in their Place: Participatory Research in Environmental and Occupational Health." Highlander Center Working Paper, New Market, Tennessee.

Merschrod, Kris. "Participation in Program Evaluation at the Regional Level in Honduras." Rural Development Participation Review. Vol. 11: 18-22, 1980.

Marsden, David; Oakley Peter; Pratt, Brian. Measuring the Process: Guidelines for Evaluating Social Development. Oxford, 1994.

Mwelekeo Wa NGO (MWENGO) and Partnership Africa Canada. Impact Evaluation A New Path for Developing an Equitable, Participatory, Sustainable Approach to Evaluating for Impacts. Report of the Process and Outputs of a Consultation on Impact Evaluation, February 20-25, 1995, Maseru, Lesotho.

Narayan, Deepa. Participatory Evaluation Tools for Managing Change in Water and Sanitation. The World Bank Technical Paper Number 207, Washington, D.C. 1993.

Oakley, Peter. "Evaluating Social Development: How Much or How Good?." IDEA: Reading Rural Development Communications Bulletin. London, University of Reading Agricultural and Extension and Rural Development Centre. Vol. 14: 12-17, 1982.

_____________. "Conceptual Problems of the Monitoring and Evaluation of Qualitative Objectives of Rural Development." Community Development Journal. Vol. 23: 3-11. January 1988.

_____________. The Monitoring and Evaluation of Participation in Rural Development. FAO, Rome, 1988.

Pagaduan, Maureen; Ferrer, Elmer M. "Working as Equals-Towards a Community Based Evaluation System. Community Development Journal.. Vol. 18: 146-159, 1983.

Participatory Research Network. Report on International Forum on Participatory Evaluation. International Council for Adult Education, New Delhi, India, 82 pp., 1988.

Pfohl, Jacob. Participatory Evaluation. A Users Guide., Pact Publications, New York, 1986.

Quinn Patton, Michael. "Developmental Evaluation." Evaluation Practice, Vol. 15, No.3, 1994. pp. 311-319.

___________________. How to Use Qualitative Methods in Evaluation. Newbury Park, Sage Publications, 1987.

____________________. Culture and Evaluation. San Francisco, Jossey-Bass, 1985.

____________________. Qualitative Evaluation Methods. Beverly Hills, Sage Publications, 1980.

____________________. Utilized-Focus Evaluation. Beverly Hills, Sage Publications, 1978.

____________________. Alternative Evaluation Research Paradigm. Frank Forks, North Dakota, University of North Dakota, February 1965.

Robinson, Sheila; Cox, Philip. Process Evaluation in Nepal: Tracking Capacity-Building in Health Development. Dialogue on Development, Technical Paper No. TP95/1, Division of International Development, International Centre, The University of Calgary, Canada, 1995.

Rudqvist, Anders; Woodford-Berger, Prudence. "Evaluation and Participation An Approach Paper (Revised)." DAC Expert Group on Aid Evaluation. Department of Social Anthropology, Development Studies Unit, Stockholm University, September 1995.

Rugh, Jim. Self-Evaluation Ideas for Participatory Evaluation of Rural Community Development Projects. A World Neighbors Publication, Oklahoma, 1992.

Salmen, Lawrence F. Beneficiary Assessment-An Approach Described. Environment Department Papers No.023, The World Bank, Washington, D.C., July 1995.

__________________. Listen to the People-Participant-Observer Evaluation of Development Projects. The World Bank, Washington, D.C., 1987.

Srinivasan, Viji. "The Methodology of Participatory Evaluation." In W. Fernandes and R. Tandon (Eds.) . Participatory Research and Evaluation: Experiments in Research as a Process of Liberation. New Delhi, Indian Social Sciences Institute, 1981.

Stecker, Carl. Participatory Evaluation: A Review of the Literature and Its Implications for Participatory Community Development, Comprehensive Paper One, University of Massachusetts, School of Education, April 1993.

Swiss Directorate for Development Cooperation and Humanitarian AID (SDC). Mirror, Mirror on the Wall....Self-Evaluation in Development Cooperation. Evaluation Service. Bern, August 1991.

Tandon, Rajesh; Fernandes, Walter. Report of International Forum on Participatory Evaluation 1-5 March 1988, New Delhi. New Delhi, 1988.

__________________________. Participatory Evaluation: Issues and Concerns. Society for Participatory Research In India. New Delhi, India. 1988.

__________________________. Participatory Research and Evaluation: Experiments in Research in Research as a Process of Liberation. Indian Social Institute. New Delhi, India, 1982.

__________________________. Participatory Research and Evaluation. Indian Social Institute, New Delhi, 1981.

UNDP. Who are the Question-makers? A Participatory Evaluation Handbook. OESP, UNDP, New York, 1997.

__________________________. "Money and Mambas or Listening to the People-A Case Study of a Participatory Evaluation of a Water and Sanitation Project." Guide to Facilitator/Resource Person, Sections A,B,C.

Uphoff, Norman. "Monitoring and Evaluating Popular Participation in World Bank-Assisted Projects." in. B. Bhatnagar and A. C. Williams, Participatory Development and the World Bank: Potential Directions for Change, World Bank Discussion Paper No. 183, World Bank, Washington D.C., 1992.

_______________. "A Field Methodology for Participatory Self-Evaluation." Community Development Journal, vol. 26, No. 4, 1991.

_______________. "Participatory Evaluation of Rural Development Projects", The International Fund for Agricultural Development, Monitoring and Evaluation Division, Economic and Planning Department, 1991.

_______________. A Field Methodology for Participatory Self-Evaluation of P.P.P. Group and Inter-Group Association Performance. Prepared for the PPP Group at FAO, May 1989.

World Bank. Evaluation and Development, Proceedings of the 1994 World Bank Conference. World Bank Operations Evaluation Department.

Yacoob, May; Cook, Tom. "Evaluating Community Participation." WASH Working Paper, N.98, Rosslyn, Virginia, USA, 1991.