Participatory Project Design: Its Implications for Evaluation. A Case Study from El Salvador

By Françoise Coupal, November 1995

Background

El Salvador is a small, yet mountainous country nestled between Costa Rica and Guatemala. It is a country that is emerging from more than a decade of civil war which has ravaged its infrastructure, left thousands of women and children to fend for themselves, undermined its productive base and left thousands impoverished. Social conditions in this country are among the worst in the region. With a population of 5.3 million persons, poverty affects 70% of the population with women and children carrying a large burden. In the large cities, it is estimated that 40% of women are single heads of household. Over one million children have no access to any type of childcare or educational services with 30% of children suffering from high rates of malnutrition.

In this period of transition and reconstruction, the Government launched in 1992 a National Plan for Reconstruction and the Alleviation of Poverty. Among the many initiatives seeking donor support are the creation of "hogares comunitarios" or daycare centres for malnourished children between the ages of 2 and 6 years of age. By 1994, the Government had established 57 daycare centres in 9 departments of the country in municipalities most affected by the civil war. Hoping to expand their program, the Government solicited technical assistance to help them design a project for the consideration by one of the major donors.

The Experience to Date

Three missions were conducted to El Salvador to prepare a project document. The consultant used a highly interactive and participatory approach which involved all levels of decision-making: senior levels of management, the technical staff, social workers, the community, daycare helpers and parents. Separate meetings were held with NGOs and the private sector to validate the daycare model and approach being used. From the onset, the project design was based on three principles: sustainability, participation and a multiplier effect.

As will be explained later, sustainability was essential to ensure that the daycare model being developed could be financially sustainable in three to four years time. This meant State support would be phased out over a three to four year period to a minimum level of support: food assistance and the salary of a daycare worker. Other items such as the maintenance of the daycare centre, educational materials and toys would be eventually provided by the parents. This required strong community participation. Participation was also essential to ensure that the project design reflected the needs and potential of the Salvadoreans. Finally, as funds were limited to about $ 5 million, the project would pilot three types of daycare models: daycare centres for the marginalized zones of San Salvador, daycare centres on-site with industry and daycare centres in the marginalized urban and peri-urban departments of Usulutan and San Miguel. Donors funds would act as seed money to attract broader support once the pilot cases were launched, tested and refined.

At the time of the first mission in August of 1993, the Government had already begun to establish 27 day care centres in 9 provinces of the country. The program was first initiated by the National Secretary for the Family and later transferred to the Salvadorean Institute for the Protection of Minors. The project team overseeing the daycare program included a Project Director, a coordinator, a psychologist, a nutritionist and 12 social workers. Each daycare centre was run by two "madres cuidadoras" or daycare workers and one helper.

The daycare centre provided children two to six years of age with early childhood care and stimulation as well as a snack and a meal. The food was provided through the country's food assistance program. A daily rate of 25 centavos or 54 was charged for leaving a child at the daycare centre.

Overall, the daycare centres were running with a mixed degree of success. The main obstacles were: a top-heavy and centralized bureaucracy, limited economies of scale, and weak community and parental participation. As the daycare centres were State supported, staff were often caught up in paperwork particularly with regard to collecting and accounting for expenses. This meant that social workers spent an exorbitant amount of time supervising the weekly accounts of each day care centre vs teaching the daycare workers about early childhood simulation or the program of activities. Rigid rules related to purchasing meant that purchases were made in the formal sector where official receipts would be given. This also meant that costs were higher by as much as three times, as compared to the informal sector. As the Government had targeted those municipalities most affected by the Civil War, distances between day care centres were large, making economies of scale difficult and reducing the overall number of daycare centres that could be supervised by a social worker (about four daycare centres a week per social worker). Overall, community and parental participation was weak or non-existent. Parents saw the daycare centres as a place to park their children. Contributions of 25 centavos, more symbolic than real, were not enforced regularly. All of these factors meant that the daycare centres were not financially viable nor sustainable without significant and continued State support.

Steps in Participatory Project Design

A number of steps were undertaken in the project design. These are:

1. Collecting the data

Basic baseline data is essential to any project: for project design, for monitoring progress or measuring the impact of a project. There existed major gaps in the types of information maintained by the existing project. As the project did not maintain basic data, it was necessary to design a simple questionnaire that could be completed by each daycare centre. The social workers were responsible for coordinating the questionnaire in the field with the "madres cuidadoras". The questionnaires solicited basic information concerning the children attending the day care centres, their families and the community. Questions included specific questions concerning the children and broader information about their families: the status of mothers (married, single heads of household), employability, average family income, housing, etc. Such information was essential. If donors want to measure the indirect benefits of placing children in daycare then having information on income or women's work status is very important. It was also very important to have a knowledge base of the community organizations and NGOs working in complementary or parallel activities that could reinforce the activities of the daycare centre.

The results of the questionnaires were incorporated into the project design. Had time permitted, the questionnaire results could have been analyzed with the project team with the view of establishing a practice of collecting baseline data. This would have also given the project team a better grasp of the users and their families. For example, in some communities, it was found that the daycare tended to cater more to families who were well off rather than to marginalized children. The data also highlighted the importance of locating daycare centres closer to marginalized zones in order to adequately access these groups. Another tool that could have been employed had time and resources permitted, could have been mapping which would have involved parents, the community, madres cuidadoras and the social workers in better understanding the resources of their community, their priorities and special needs.

From the beginning it was assumed that the daycare centres in and by itself would not alleviate poverty or its conditions. It was important for the daycare centres to maximize their impact by complementing other development initiatives. Thus, for example the project should seek to take advantage of any primary health care projects in the region to ensure that all children enrolled in the daycare centre would receive vaccinations and receive medical check-ups on a regular basis. Water sanitation projects were important in lowering the rate of diarrheal diseases and other viruses. Credit programs could assist mothers to establish small businesses since their time was freed up as a result of the daycare centres.

2. Visiting the Field

Field visits were made with the project staff to ascertain the current status of the daycare centres and the model that should be supported in the future. Private daycare centres were also visited providing an excellent point of comparison with the daycare centres supported by the State. For example, there were many examples of daycare centres which integrated mothers into the daily routine of the daycare centres (this encouraged a transfer of knowledge between what was being taught at the daycare centre to the home), where a daily fee was charged representing about 10% of the daily wage of the family, or daycare centres which used locally made products, thus reducing substantially the costs. Visits were also made to the provincial health and education ministries to discuss possible areas of collaboration. Integrating staff into the field visits was essential as it allowed them to expand their horizons and witness for themselves other daycare models. Contacts made at the provincial level also expanded their possible networks.

3. Defining the Logical Framework Analysis

Most donors now use what is called a logical framework analysis to summarize the activities, results and verifiable indicators related to each objective of the project. The framework provides in a nutshell, the core activities and outcomes of a project. It also provides the key indicators for monitoring and measuring the results of the project. Customarily, this exercise is undertaken by the person responsible for the project design with some type of input from the field.

However, it is possible for stakeholders to be the authors of the logical framework analysis. While time constraints limited the full application of this exercise, it is feasible to undertake the logical framework analysis with project or technical staff, parents and members of the community. This can be done by holding special workshops with the staff and with representatives of the community. The facilitator can provide the participants with a sample matrix of the logical framework analysis. Once the objectives have been agreed upon, staff and community representatives can work at defining the respective activities, results and performance indicators.

The advantage of having project staff and key community actors participate in the logical framework analysis is that they begin to truly visualize where the project is going, what steps are required to achieve the desired results and to integrate the concepts in their own work. In the case of El Salvador, a previous evaluation had revealed the lack of objectives and coherent activities to guide the project. Thus, such an exercise can be useful as a learning tool for staff and communities.

4. Involving NGOs in the Project Design

In the past decade, the World Bank has supported State supported daycare centres in Columbia and Bolivia. In Columbia today, there are over 33,000 daycare centres throughout the country. In both these countries, the role of NGOs in the delivery of services is quite limited despite their expertise at working at the grass-roots level.

From the beginning, the project sought to integrate NGOs in the delivery of services. Working with NGOs had a number of advantages. First, NGOs had the greatest experience in working with the poor and marginalized populations of the country. NGOs knew that working with poor communities meant working evenings or on weekends--practices that were uncommon with Government workers who were use to working nine to five. The use of participatory and informal approaches critical to the success of the project was also a general characteristic of NGOs as compared to Government. Secondly, NGOs had access to all parts of the country which was important since, during the Civil War, the government was unable to penetrate certain regions of the country. Thirdly, NGOs offered more flexible structures, thus reducing the amount of bureaucracy and redtape. This meant that the role of the State would be more at the policy level and in supervising the delivery of services of NGOs rather than in actual execution.

From the onset, NGOs were invited to comment on the project design of the project and helped to refine the daycare model being proposed. In the project design, NGO participation in the execution of the project would be limited to three individual organizations or consortiums. Greater NGO participation in the delivery of services could be contemplated in future phases of the program after a sustainable daycare model had been tested and refined. First, it was necessary to develop the model, define procedures and standards that could then be replicated on a much larger scale.

NGO comments were essential to the project design. Some of their comments highlighted the limited capacity of communities to assume all the financial responsibilities or burden of the daycare centres. NGOs felt this should be share with the State. Most wanted an arms length relationship with the Government for the delivery of services. The importance of self-management and community participation in the running of the daycare centre was also stressed by the NGOs.

5. Involving the Private Sector

While the majority of Salvadorean women work in the informal sector, a growing number have been drawn into the "maquila" or the textile industry especially in San Salvador. This industry currently employs over 35,000 women. Furthermore, it is estimated that absenteeism in the workplace related to childrearing accounts for up to 20% annually. This figure represents a tremendous loss to industry. Turnover rates are also high. It is also not unusual for women to leave children in the care of older siblings, who may be as young as four to five years of age, or have children locked-up in the home.

A portion of the project was thus targeted to the private sector with companies that employed a high percentage of women. From the onset, it was expected that the private sector would commit a portion of their funds to cover administration and salaries related to the daycare centres. The role of the State would be to provide technical assistance and training to the staff of the daycare centres. To help identify companies, the Salvadorean Association of Industrialists was approached to help identify small and medium-size industrialists who could be considered progressive with a social vision of business. By focusing on leaders, it was hoped that a degree of support and enthusiasm for this initiative would be generated and, as a result, attract more conservative employers.

As members of the private sector pointed out, the daycare model used alongside industry had to be adjusted to the needs of industry. For example, women with children 3 months-2 years of age were more likely to drop out of the workforce or require more leave days. It was thus critical that the daycare centre be open to this age group. The hours of the daycare would also have to be adjusted to the work hours of the worker. The size of the daycare centre in most cases would also have to be expanded to 60 children vs 25 children in the other daycare centres due to the high needs of industry.

Interest in the project grew, and at each meeting more employers interested in participating in the project arrived at the meetings. At these meetings, each employer shared with the others the profile of their company, the needs of women workers, and suggestions for a possible daycare model with the private sector.

Involving local community leaders, NGOs and the private sector contributed significantly to the project design. It meant that the daycare model being proposed was more realistic and could build on earlier efforts. It was also possible to witness the demand and need for such services.

6. Validating the Project Design: Workshops with Stakeholders

The above sections highlighted the ways in which the NGOs and the private sector were part of the project design. At every stage of the project design, workshops were also held with all levels of the National Institute for the Protection of Minors which would be involved in the project. The workshops sought to elicit feedback on the project design and ensure that the objectives, activities and outcomes anticipated were realistic. At least once to twice a week, workshops were held with certain levels of the project staff (ie. technical) and the entire project staff including management.

These workshops provided staff with the opportunity to comment on the project design. Holding separate workshops with technical staff was important as they felt more comfortable expressing themselves. Having workshops with senior management was also important to ensure that the project design was on track and that senior management was in agreement.

In the case of El Salvador where there was not a base project document to work from, it became essential to use participatory approaches that were highly interactive in order to flesh out the design of the project.

Benefits to Using a Participatory Approach

During the third visit to El Salvador, the author witnessed significant changes in the existing project. To an outsider, these may seem modest, but to the project they symbolized change and improvements. One of the important changes occurred with the staff. The original group of social workers came from San Salvador. This meant that the social workers had to travel long distances every day to visit the daycare centres. In fact, the majority of time was spent travelling to and from the daycare centres. As the social workers were not from the regions, they had a limited understanding of the community environment and dynamics. These limitations emerged through discussions with staff. As the project grew, the social workers were hired from the region where they lived and resided.

Some daycare centres began purchasing in the informal sector and setting up bank accounts for their daycare centres. The possibility to purchase in the informal sector represented significant savings for the daycare centre and also contributed, albeit in a modest way, to supporting the local economy. The establishment of bank accounts, critical in the project design in order to encourage greater self-sufficiency for the communities, meant that communities could exert greater independence and learn to manage their funds.

Another benefit arose from the meetings held with the NGOs. The National Institute for the Protection of Minors had been approached by an NGO to supervise and create about 10 daycare centres in the region where they worked. Rather than have the Institute administer the daycare centres, the latter would be under the guidance of the NGO. This delegation of authority is important and represents an opportunity for the NGO to begin working more in tandem with the Government.

The project team had also began working more closely with the communities to solicit greater participation with regard to the daycare centres. Daycare committees which were established upon the creation of the daycare were revitalized and members trained to take on various responsibilities: the selection of the daycare workers, administration of the daycare, resolving disputes, etc.

Lessons Learned and Their Implications for Undertaking Participatory Evaluations

There are always many lessons that can be learned in an exercise of this nature. While it is outside the scope of this article to numerate all the lessons learned, there are a number of lessons and steps that can be drawn from this case that would be of relevance for undertaking participatory evaluations. These are:

• it is critical that the donor and project recipients understand what participatory approaches entail. Participatory approaches should be based on transparency and the sharing of information. This may be at odds with donors who may not want to share all the information with the project such as the terms of reference for an exercise or the results of an evaluation. Workshops and full briefings with the stakeholders are critical at all stages of the evaluation: in the design of the terms of reference, in determining the methodology to be used, in the analysis of the data and in proposing suggestions for the future.
• The donor's role should be at arm's length. Those who are central to project design and participatory evaluations are the stakeholders themselves. The donor should be there to listen, observe and learn and not to dictate their own institutional priorities.
• Participatory approaches are more labour-intensive than traditional project design or formal evaluation exercises. Sufficient time and resources should be foreseen for defining the terms of reference, collecting and analyzing the data, organizing workshops with staff and the communities and preparing the report.
• Difficulties may arise trying to use participatory evaluation techniques with projects that have not been designed or executed in a participatory manner. It is thus important that all parties are aware of the methodology and its implications.

Participatory Evaluations

There are a number of key steps when undertaking a participatory evaluation that vary from more formal evaluations. These are:

1. The socio-economic context. Participatory approaches do not operate in a vacuum. Understanding the socio-political context is important. For example, undertaking a participatory project design or evaluation in Nicaragua vs El Salvador would be quite different. With the revolution, there has been a longer history of community action and critical analysis in Nicaragua. In El Salvador, socio-economic ties have been shaped by socio-economic inequities and a degree of paternalism that is pervasive throughout the society. These differences can affect the degree of participation, openness and questioning by project staff and recipients.

   Participatory approaches involve crossing class lines. Where strong class divisions exists, using a participatory approach can be challenging or even controversial. It is important that project recipients understand the importance of being inclusive and having all levels of decision-making participate.

2. The current experience of stakeholders. This becomes an important starting point with the stakeholders. It is important to review from the beginning the experience stakeholders and beneficiaries have had with the project, the accomplishments and impact of the project, and the key constraints. The facilitator must be able to listen, assist stakeholders in asking the key questions and move the discussion along when necessary. It is important to review the methodology with the recipients and to make sure that all understand their respective roles.
3. The evaluation design. In contrast to formal evaluations where the evaluation design is defined by the Donor in isolation or with some input from the project, participatory evaluations must involve the stakeholders in the evaluation design itself. The stakeholder plays a central role in setting the objectives of the evaluation, in defining the key questions to be asked and the methodology and verifiable indicators to be used.

4. Data Collection and Consolidation. It is important that the stakeholders play a key role in the data collection and its analysis. Deciding on the appropriate methods will vary from group to group and might include: questionnaires, mapping, transects, interviews, informal discussion meetings, sampling or case studies. The use of METAPLAN techniques, a methodology designed by the Germans and perfected in Latin America which involves the use of colored cards to organize participants ideas is another tool that can be effective.

   Where groups have a low level of literacy, one may need to use other techniques than formal data collection methods such as mapping, folklore, songs, or theatre to evaluate activities so that stakeholders are not left out.

5. Analysis. While generally every stage of participatory evaluations involve some type of workshop, the analysis of the data collection and the findings are critical. It is important that the stakeholders are involved in the analysis of the data and have an understanding of the findings. The analysis and recommendations made by the participants can then be consolidated into the final evaluation report.

   Group meetings and workshops with all levels of the project staff are essential to validate the project design. It is important to ensure that key staff participate throughout the different workshops to ensure the same information base and enable a better understanding of the project activities and its potential. These workshops should also be broadbased and include any outside stakeholders that are or will be part of the project such as NGOs or the private sector.

6. Feedback and Using Evaluation Findings. Stakeholders must have an opportunity to comment on the final report. In fact, the report must be a reflection of what came out of the various workshops: the key findings, recommendations and a future plan of action. A good evaluation should provide stakeholders with concrete tools and recommendations for stakeholders to reorient the project with or without donor funding.

What should become apparent in the above steps, is that the role of the "evaluator" dramatically changes to that of a facilitator. It is the participants themselves that are central to the process. Whereas in traditional evaluations the stakeholders are most often passive participants supplying information as needed, in participatory evaluations they become central to the process-- in other words active participants.

Differences with More Traditional Approaches

What becomes apparent with participatory approaches are its differences with more traditional approaches. More traditional evaluations tend to be more donor focused and linear. The donors are the key clients providing financial support and defining the terms of reference for the evaluation. Evaluations are typically used to fulfill an institutional need or programming activity than for the project or recipient themselves. The recipients are, in most cases, passive providing information but not participating in the evaluation itself.

In contrast, participatory approaches places as much emphasis on the process as on the final output: the report. While donors do provide the funding, they are not central to the process. The stakeholders are. They are the ones that define the terms of reference, undertake the data collection and analysis and provide recommendations. Whereas in formal evaluations, the stakeholders provide information, with participatory evaluations they are the key agents. Thus, the "evaluator" becomes in actual fact the facilitator animating workshops, guiding the process at critical junctures and consolidating the final report, if necessary, based on the findings of the stakeholders.

While participatory evaluations pose its own challenges, like project design, they have the capacity to empower the recipients. The active participation of stakeholders can result in new knowledge or a better understanding of their environment. It is this new knowledge and understanding that can enable stakeholders to make changes they themselves have discovered or advocated. Stakeholders feel a sense of ownership over the results which is not coming from an outsider nor a donor.

Conclusion

This article has presented a case study of a project designed in El Salvador using participatory approaches and techniques. Many of the approaches and techniques used are not specific to the case study nor to project design.

Participatory techniques and approaches can be used at different stages of a project: at the design stage, in its execution, while monitoring the project or evaluating its overall impact. However, evaluations using participatory techniques can be more effective where participatory approaches have been used from the onset and are part of the project's ethos.

While participatory techniques may vary, what is key is that the stakeholders are central to the process. The "evaluator" becomes a facilitator, someone who listens and at times helps stakeholders to ask key questions. It is the stakeholders, however, who are the authors of change.

Seen in this way, participatory evaluations can be much more dynamic. By being part of the process, stakeholders can implement the changes that make sense to them. This is in stark contrast to more traditional approaches, where a final report is submitted and most often than not read by only a few.

Sound development must involve those that are central to the development process. Using participatory approaches and techniques in areas such as project design and evaluation, represent significant improvements in more traditional project design and evaluation methods. Participatory techniques and methods aim to more comprehensively incorporate the feedback of a wide-range of stakeholders who are central to development efforts.

Bibliography

Colletta, Nat; Perkins, Gillian. "Participation in Education". Environmentally Sustainable Development, Participation Series, Paper No.001, February 1995, The World Bank.

Dinámica Integral Ltd. "El METAPLAN. Metodología eficiente y eficaz para lograr objetivos con participación grupal. Mejoramiento y desarrollo permanente". Colombia.

Eming Young, Mary. "Integrated Early Child Development-Challenges and Opportunities". World Bank internal paper, 1994.

Freedman, Jim. Participatory Evaluations. A Practical Alternative to Evaluating Rural Development Projects. Audit and Evaluation Division, CIDA, Hull, Quebec. 1991.

Feuerstein, Marie-Therese. "Finding the Methods to Fit the People: Training for Participatory Evaluation". Community Development Journal, Volume 23, No. 1, 1988.

Lawrence, John. "Engaging Recipients in Development Evaluation. The "Stakeholder Approach". Evaluation Review, June 1989, pp. 243-256.

Ministry of Planning and Coordination of Economic and Social Development. "Consolidating the Peace through National Reconstruction and Poverty Alleviation". Report to the Consultative Group Meeting, Paris, April 1, 1993.

Narayan, Deepa. "The Contribution of People's Participation. Evidence from 121 Rural Water Supply Projects". Environmentally Sustainable Development Occasional Paper Series No. 1, 1995, The World Bank.

_____________."Participatory Evaluation. Tools for Managing Change in Water and Sanitation". World Bank Technical Paper Number 207, 1993.

PACT. "Participatory Evaluation. A Users Guide". Private Agencies Collaborating Together, July 1986, New York.

Rugh, Jim. Self-Evaluation. Ideas for Participatory Evaluation of Rural Community Development Projects. World Neighbors Publication, Oklahoma, 1992.

_______________. The Contribution of People's Participation. Evidence from 121 Rural Water Supply Projects. Environmentally Sustainable Development Occasional Paper Series No.1., July 1995, The World Bank.

Salmen, Lawrence. "Beneficiary Assessment. An Approach Described." Environmentally Sustainable Development, Social Assessment Series, Paper No.023, July 1995, The World Bank .

_______________. Listening to the People. Participant-Observer Evaluation of Development Projects. 1987, The World Bank, Oxford University Press.

UNICEF. Una agenda en favor de la infancia y la mujer de El Salvador. UNICEF, El Salvador, febrero 1994.

____________. Nuestra niñez, nuestras mujeres, nuestro futuro. UNICEF, El Salvador, febrero 1994.

____________. Early Childhood Development. The Challenge and the Opportunity. UNICEF, New York, March 1993.

____________. "Análisis de situación de la infancia y la mujer. El Salvador". UNICEF, El Salvador, octubre 1991.

Uphoff, Norman. "A Field Methodology for Participatory Self-Evaluation". Community Development Journal, Volume 26, No.4 1991.

The World Bank. World Bank Participation Sourcebook. Environmentally Sustainable Development, June 1995, The World Bank.

_____________. The World Bank and Participation. The World Bank Operations Policy Department, September 1994, The World Bank.

_____________. Staff Appraisal Report, Bolivia, Integrated Child Development Project. Country Department III. Human Resources Operations Division, Latin America and the Caribbean Region, The World Bank, June 7, 1993

______________. Staff Appraisal Report, Colombia, Community Child Care and Nutrition Project. Country Department III. Human Resources Operations Division, Latin America and the Caribbean Region, The World Bank, May 2, 1990.

About the Author

Françoise Coupal is founder of Mosaic International, a private consulting firm specializing in participatory development. Ms. Coupal has worked with an array of stakeholders: the private sector, communities, Government, NGOs, and education institutions and applied participatory techniques to project design, evaluation, strategic planning exercises and in facilitating workshops. Ms. Coupal started her professional career working with a Canadian NGO and then worked with the UNDP in Costa Rica and Niger, and the International Development Research Centre based in Ottawa. Having lived and worked in Latin America, Africa and Asia over the past 15 years, Ms. Coupal employs alternative approaches and seeks innovative solutions to development issues at hand. Clients of Mosaic International include at the international level: The World Bank, the OAS, the IDB; and on the Canadian front such organizations as CIDA, CBIE, AUCC and private sector companies.

Ms. Coupal received her Bachelor of Arts degree from UCLA in 1979 and continued her graduate studies at the Johns Hopkins School of Advanced International Studies (SAIS) and the Norman Paterson School of International Studies at Carleton University resulting in a Master of Arts degree in Development Studies in 1984.

If you have any comments or questions, please feel free to contact Françoise Coupal at coupal@mosaic-net-intl.ca.