Evaluations that make a difference is a collection of 8 evaluation stories from around the world which is one of the first pieces of systematic research looking at factors that contribute to high quality evaluations that are used by stakeholders to improve programs and improve people’s lives. This initiative collected stories about evaluations that made a difference, not only from the perspective of the evaluators but also from the commissioners and users. The stories in this collection tell powerful stories about the findings in the evaluations and the ways the evaluations contributed to the impact of the programs. You may access the report and all the stories here, in English, Spanish, and French.
In these weekly posts, we will be sharing each story… Comments are very welcome!!
If you don’t ask, you won’t see it!
Contemplating the indigenous women of the Sierra Tarahumara, a mountain area in the state of Chihuahua in northern Mexico, evokes bucolic scenes from the time of the Spanish conquest. These communities have maintained themselves for centuries far from cities, clustered in small villages and scattered family groups, cultivating a few seasonal crops and raising chickens, goats, and cattle. Almost all live in poverty, and in many cases they are semi-nomadic.
Access to the Sierra Tarahumara is difficult. Sometimes it takes several days to reach the settlements. It is therefore quite difficult to make contact with them to interview and select candidate families for the Programa de Desarrollo Humano Oportunidades (Human Development Opportunities Programme, hereafter referred to as Oportunidades). But the effort is well worth it because the programme is quite remarkable.
This programme, implemented in Mexico since 1997, aims at nothing less than breaking the seemingly endless cycle of poverty that typifies many rural communities. It does so by using a conditional cash transfer (CCT) approach, where families are provided with payments that are conditional upon undertaking certain activities, such as ensuring regular attendance of their children in school or obtaining certain health services. These incentives assist in the achievement of higher standards of education, health, and nutrition, and also provide necessary support to the people of the Sierra Tarahumara to undertake economic activities that enable them to increase their family income and quality of life.
The indigenous communities were among the intended beneficiaries of the programme since its inception, and they also participated in their evaluations from 1999 to 2006. But although these evaluations confirmed that indigenous people were effectively participating in the programme, it was unclear if they were achieving the stated objectives in terms of health, education, and ultimately, economic performance. According to Mercedes Gonzalez de la Rocha, anthropologist and head of the qualitative impact evaluation of Oportunidades, the absence of a clear focus on indigenous communities had left a blind spot in the programme’s knowledge base. Or as she puts it, If you don’t ask, you won’t see it!
This statement constituted a starting point for raising the influence of an ethnicity variable in the new qualitative evaluation to be conducted in 2008. By then, the programme had a decade of experience in implementation in Mexico, which allowed for a thorough evaluation. Evaluators designed a strategy for field work comprising 11 indigenous intercultural regions in the states of Chiapas, Chihuahua, Oaxaca, and Sonora. In each location, the programme’s coverage and operations were analysed. The evaluation identified the main obstacles to programme implementation with particular attention to the relationship between the extensionists (promotores) and the women representing the indigenous communities, the vocales.
What they found was quite unexpected: there were serious communication problems with language. Virtually none of the promotores and only a few of the vocales were bilingual. Although the programme had been operating since 1997 with more impact in indigenous areas than in non-indigenous areas (as had been documented by previous evaluations), the important problem of communication had not been sufficiently addressed. For instance, the evaluation found that young indigenous women – most of whom were bilingual, although their Spanish was often limited – did not accurately understand the Oportunidades employees and the technical information they provided. This situation was exacerbated in the case of elderly women, who spoke no Spanish at all. The promotores assigned to the region did their best to overcome the language barrier, but the results were unsatisfactory.
In some areas, the majority of indigenous women did not understand what the programme was for. They couldn’t understand what good it did to spend hours listening to medical specialists who spoke about issues they could barely comprehend. Sometimes the information that was being communicated conflicted with their traditional customs. For example, when indigenous women participated in training on the importance of a physical examination for possible breast cancer, it was clear that the idea of a stranger touching them in their private parts made them very uncomfortable. These women do not even undress in front of their husbands! Thus, a practice intended to save their lives was totally unacceptable for cultural reasons.
The evaluation found that although the programme’s coverage had improved in some indigenous areas, in the Sierra Tarahumara, a whopping 30% of the population remained outside the programme. It was clear that the lack of access to health services and education was brought about by the problem of monolingual families who could not benefit from the written and oral information conveyed by Oportunidades staff members to the women.
It was urgent to take measures to solve this serious problem, and Oportunidades did just that. The qualitative evaluation suggested that bilingual promotores be recruited from indigenous youth alumni so as to contribute to better communications and smoother operations in indigenous communities. This arrangement also led to a further positive impact by providing jobs and wages to the young in their own living space, allowing them to share the learning they had acquired right there in their own communities.
The then general coordinator of Oportunidades, Dr Salvador Escobedo, recalls:
We managed to make the change in rules of operation to include bilingual extensionists, and that was the first step. Then in parallel we ran a training programme with the National Institute of Indigenous Languages (INALI) in order to generate a mechanism to evaluate the indigenous youths we wanted to hire as extensionists who speak an indigenous language. We have sought to work with the 13 most spoken languages such as Tzotzil, Maya, and Totonac.
To do this, INALI began training a group of 15 young men and women with diplomas as trainers of social programmes in indigenous languages. They were the first to reach the remotest areas of the country to carry the voice of the programme to where it was most needed. Says Escobedo, ‘The project trained 350 promotores in order to achieve almost total coverage of the monolingual indigenous populations in Yucatan, Oaxaca, Chiapas, Jalisco, and the Sierra Tarahumara. By the time I left the programme, we had trained a total of 250 extensionists, and awaited evaluation on the success of this implementation.’
The process of bringing bilingual promotores into the programme unfolded steadily over the next 2 years. As their understanding increased about the importance of sending their children to school, feeding them properly, and learning how to use resources, indigenous women became increasingly willing to participate and interact with the promotores and with each other. In some areas they even formed groups for early child education. The role played by the qualitative evaluation with its remit to address the marginalisation caused by monolingualism was key to this transformation.
Another important aspect that was pointed out by the qualitative evaluation of 2008 concerned the inefficient (and even absurd) survey of each household to assess whether or not it was eligible for the programme. In the case of the communities of the Sierra Tarahumara, this was redundant because every household was unquestionably poor – not to mention the difficulty and cost of getting there just to conduct the survey.
Escobedo, who understood the realities of indigenous communities in Mexico, supported the proposals arising from the qualitative evaluation that were eventually endorsed by the president.
The first step was to convince Oportunidades collaborators such as government officials and international organisations like the Inter-American Development Bank and the World Bank of the need to change the rules for coverage in isolated areas. That involved a major challenge for the programme managers because, beyond being convinced of the real value of this recommendation, they needed to effect those changes without appearing to refute the spirit of Oportunidades, which was historically focused on the poor while requiring certain conditionalities of the beneficiaries such as health checks and school attendance.
In late 2011 and early 2012, a severe drought hit the Sierra Tarahumara. Rumours were rife about indigenous people committing suicide for lack of food, although these were subsequently found to be false. Dr Iliana Yaschine, former director of evaluation of the programme from 2002 to 2006, coordinated a study documenting the work of Oportunidades in the Sierra Tarahumara during the drought crisis, recalls:
That report was published in a Chihuahua newspaper and generated an immediate response from the federal government, which decided to intervene in various regions with the Secretary of Social Development. It was then that they communicated the decision that Oportunidades should intervene to solve the problem precisely in the area of the Sierra Tarahumara.
The important mobilisation and attention to the area made it possible to detect 8,000 families (40,000 indigenous people) who had been dropped from the programme because they had not fulfilled such co-responsibilities as sending children to school or attending health talks. The figure was worrisome, and returning these people to the programme would be difficult, if not impossible, given the rules of operation. Reaffirming this contention, Escobedo asserted:
The World Bank and Inter-American Development Bank opposed eliminating the requirement of the co-responsibilities in these areas, in part because they would undermine the programme, thus generating a new programme. That´s the reason why it could not be done overnight. It is sad and painful, but very real.
However, given the magnitude of the problem in the context of the severe drought affecting indigenous people in the Sierra Tarahumara, the situation was resolved after the intervention of the President Felipe Calderón himself, who ordered the immediate reinstatement of 8,000 families.
This was accomplished by integrating the reinstatement to the Food Support Program, a transfer programme that did not require the fulfillment of co-responsibilities that was also operated by Oportunidades. The indigenous promotores brought the good news to their communities, while a renewed training of bilingual extensionists was encouraged to address immediately the indigenous families.
In addition, the rules of the Oportunidades programme were changed to implement the strategy of full coverage (as it had been recommended by the qualitative evaluation). This allowed incorporating families into the programme from small and distant locations without undertaking a home survey. This change helped broaden the programme’s coverage and, in addition to the reincorporation of the families mentioned above, it made serving the entire indigenous population of the Sierra Tarahumara possible.
According to Yaschine:
Without the qualitative evaluation, it would have been impossible to make that change, even with the drought crisis. That is my perception. I observed the implementation of this decision in the field at a time of crisis. Special efforts were made to deploy support for the affected families, not at the household level but in the care centres where groceries were delivered. I remember what the qualitative evaluation said on this subject, and how that had prompted the changes during the crisis.
The findings and recommendations of the qualitative evaluation in the indigenous communities provided crucial information for decision making. Managers made changes that impacted the participants of the programme directly. In the case of the bilingual promotores, the evaluation noted a dimension of cultural adaptation that had been neglected in the original design of the programme, and once it was incorporated the benefits were obvious. As for the evaluation’s strategy of full coverage in the Sierra Tarahumara, the onset of drought favoured the implementation of a recommendation for the best care of indigenous communities.
The history of the qualitative evaluation shows the relevance of cultural sensitivity to the evaluation of social projects. In the words of Gonzalez de la Rocha:
If you don’t ask, you won’t see it!
———————
Co-authors Mercedes González de la Rocha and Agustín Escobar Latapí directed the qualitative external evaluation of the programme and made the recommendations that led to improvements in beneficiaries’ lives.
Iliana Yaschine was Director of Evalua on in the programme from 2002 to 2006.
Pablo Rodríguez-Bilella and Omar Zevallos helped shape the story.
Salvador Escobedo (Program Director at the time) provided helpful insight.
Photographs were taken by Agustín Escobar Latapí and Iliana Yaschine.
Note that the programme was originally called Progresa, then Oportunidades (the name in use when this story took place), then Prospera, the present name.
Website of the University of Texas that concentrates information on external evaluations of the Oportunidades programme.
Oportunidades / Progresa official website, with plenty of general information on it.
Evaluations that Make a Difference 