Generating and Using Evidence for Program Design: Lessons from Evaluations in Pakistan and Peru

Abstract

The design and implementation of development programs is driven by a set of assumptions on development interventions that typically overlook the role of context; particularly social norms and institutions. Moreover, evaluation is still focused in understanding if an intervention works, instead on how to make it work better. This dissertation discusses the evaluation of a marketing intervention in Pakistan, and the evaluation of a participatory development strategy in Peru. Neither intervention produced the expected results. Rather than stopping there, I discuss the reasons behind their lack of effectiveness, specifically looking at elements for program re-design.

The first essay discusses the randomized evaluation of a marketing intervention in Pakistan. The hypothesis was that given the positive role models featured in brochures promoting a microfinance product, women would increase their demand for loans. The brochure, however, had a negative effect in program take up among the poorest women. The likely reason behind this: prevailing social norms regarding role of women.

The second essay stresses that the randomized evaluation experiment should not be taken as indicating that marketing is ineffective to improve the impact of microfinance in rural Pakistan, and that the role of social norms in microfinance can be internalized and used in the re-design of the brochure along several dimensions. Using theory of change and realistic evaluation approaches, I propose a framework that combines formative and process evaluation to design and pilot alternative marketing intervention in Pakistan.

The third essay features the evaluation of the participatory strategy of El Alto, in Peru. This was a study with very limited data and virtually no control of the research team over the intervention. A mix of quantitative and qualitative techniques is used for outcome and intervention evaluation, and the framework presented on the second essay is used to understand why the participatory strategy has not been successful in sustaining participation. Originally an evaluation of small pretensions, it was used as an opportunity to revisit the objectives of the strategy, improve intervention design, and establishing a monitoring system based on administrative data. A case is made for complementary, context-based interventions.