Content+1

//Slide 1://

INTEGRATING IMPACT ASSESSMENT INTO PROGRAMME DESIGN Week 1: Theories of Change and Indicators

Paul Winters Version: Oct 12, 2011


 * I. Introduction to Theories of Change:**

//Slide 2://

The first step in designing an impact assessment plan is to work with program stakeholders to determine the theory of change.

The theory of change (as defined in [|Gertler et al], 2011, pp. 22-23): Is a description of how an intervention is supposed to bring about a desired result Provides the causal logic of why the activities undertaken in a particular project, programme or policy will reach its intended outcome Is the basis for the programme logic or results chain Helps to identify the key questions that need to be answered, or equivalently the hypotheses that need to be tested, in an impact assessment

Programme designers often get fixated on the activities to be undertaken as part of a programme forgetting that these activities have a greater purpose. By working with stakeholders on the theory of change, the impact assessment team can ensure that the link between activities envisioned in the programme are logically linked to long-term outcomes and impacts. This helps not only in designing the impact assessment but in designing the programme.

//Slide 3://

In Context: Land Formalization Program in Paraguay The Government of Paraguay is currently initiating PROCAR II, a programme to formalize the national land management systems in rural areas. The objective of the programme is to improve agricultural productivity and increase farm income through improved land tenure security. PROCAR II includes four components: (i) modernization of the cadastre, (ii) modernization of the land registry, (iii) local land formalization in eight pilot districts, and (iv) the establishment and strengthening of DINACAR a new government entity to manage the national land management system. The theory of change of the programme is noted in the next slide. The key expected result for the programme is that the new management system will provide more secure rights over land being used by farmers, particularly in the eight pilot districts. This security will (i) induce more spending and investment in agriculture, (ii) allow farmers to use land as collateral and borrow for agriculture, and (iii) enhance the number of land transaction so land is used by its most efficient user. This theory of change leads to a series of evaluation questions or hypotheses that can be tested with a well designed impact assessment. In this case, the programme has been designed so that the eight municipalities will be randomly assigned to treatment and control groups. The four treatment groups will be “treated” at the initiation of the programme while the control groups will begin receiving “treatment” two years later. In the interval, an evaluation will be conducted that compares key indicators between the treatment and control groups.

It would be nice if we could find images to illustrate each of the "in context" examples.

//Slide 4:// Image also available at http://box.net/files#/files/0/f/123533917/1/f_98343739. Insert Paul's voice over explanation.

//Slide 5://

Discussion Forum

Are you currently working on a development programme that will be or is being implemented in the field? Can you describe the theory of change of that programme? Is the causal logic from activities to impact clear?


 * II: Evaluation questions and intended/unintended impacts**

//Slide 6://

The discussion of the theory of change with stakeholders should lead to a series of evaluation questions or hypotheses to be tested through an impact assessment. For the Paraguay land formalization programme noted in the previous slide the following are some of the hyoptheses that will be tested: Hypothesis 1: Land formalization will increase agricultural spending and investment Hypothesis 2: Land formalization will increase access to credit via formal channels Hypothesis 3: Land formalization will increase land sales and rentals Hypothesis 4: Land formalization will increase agricultural productivity Hypothesis 5: Land formalization will increase the value of land

//Slide 7://

These evaluation questions can include both intended and unintended impacts of the programme.

Intended impacts are those included in the results chain or logical framework and include the steps of the causal chain that bring about the long-term outcomes and impacts. These include all of the hypotheses noted for the Paraguay land formalization programme.

Unintended impacts of the programme usually are negative possible outcomes of the programme that the project team may be concerned about. They are often included as “risks” in a result chain or logical framework. Even if not included explicitly in programme documents, they should be discussed and noted in an impact assessment plan. For example, in the case of Paraguay land formalization programme some unintended consequences to be tested might be the following: Hypothesis 6: The process of land formalization increases land conflicts Hypothesis 7: Land formalization leads to greater porperty rights for men increasing their bargaining power over women in the household Hypothesis 8: Land formalization leads farmer to increase the use of highly toxic pesticides

//Slide 8://

QUESTION Which of the following would not necessarily be viewed as an unintended consequence of a programme?

A reduction in the productive use of adult labor as a results of a programme that provides cash to poor households to send their children to school. An increase in child labor in a programme that promotes microenterprises. A change in pesticide use by farmers that are attending farmer field schools to learn about integrated pest management. Degradation of pasturelands as a result of a programme that promotes ownership and sale of livestock. A decrease in wages for casual day laborers as a result of a food-for-work programme.

ANSWER The change in pesticide use is expected once farmers learn about integrated pest management. Of course, pesticide use could increase as a result of additional knowledge regarding the pests biological cycle, which may not be intended by the programme.

Slide 9:

In Context: Unintended impacts of conditional cash transfer programmes Conditional cash transfer (CCT) programmes provide cash to poor households conditional on their children regularly visiting health clinics and attending and enrolling in school. There has been a concern over providing the poor financial benefits linked to their children since Thomas Malthus provided an elegant and dismal assessment that increased support to the poor would only exacerbate their poverty by allowing them to marry earlier and hence bear more children. To test the hypothesis of unintended fertility consequences of CCT programmes, Stecklov et al (2007) analyzed the short-term effects on births and pregnancies in CCT programmes in Mexico, Nicaragua and Honduras. Each of the programmes was designed in combination with an experimental approach to impact assessment allowing for impacts to be attributable to the programme.

The analysis showed that while there were no fertility effects in Mexico and Nicaragua, in the case on Honduras the number of births and pregnancies increased as a result of the programme. The difference in results appeared to be due to the fact Honduras had an open eligibility roster while Mexico and Nicaragua had closed rosters. In Honduras, women who lived in programme communities could start receiving payments once they were three months pregnant. The Honduran government later changed their rules to close the roster and avoid this incentive.

//Slide 10://

Discussion Forum

Are you familiar with development programmes that have had unintended consequences? What were the consequences and why did they occur? Could the programme have designed in a different way to avoid those consequences?


 * III. Spillover effects:**

//Slide 11://

A final consideration in discussions of the theory of change and the determination of evaluation questions is the possibility of indirect or spillover effects.

Spillover effects refer to the indirect impacts of a programme on the non-target population—that is, the neighbors, family, friends, etc. of the programme beneficiaries. As with direct impacts, they can be intended and unintended.

Four types of spillover effects can be identified (see Angelucci and Di Maro, 2010, pp. 6-7): The paper is in the background material and should be linked there Externalities: When programme benefits are transmitted directly from beneficiaries to non-beneficiaries General equilibrium effects: When programme effects are transmitted to the local economy Interactions: When non-beneficiaries receive some advantage (or disadvantage) from interacting with beneficiaries Behavioral effects: When non-beneficiaries change their behavior as a result of observing changes in the behavior of beneficiaries

In many cases, spillover effects can be as important or more important as direct effects and should be measured. As shall be seen in next week’s lesson, they also have important implications for determining the approach to assessing impact.

//Slide 12://

QUESTION Identify the type of spillover effect in each of the following cases: A health programme alters the social norm whereby husbands oppose their spouses being screened for cervical cancer by male doctors. This creates an increase in female screening among beneficiaries as well as non-beneficiary women. An emergency seed programme provides seeds to a large number of farmers in a region thereby lowering the price of seeds in the local market. Children who receive free textbooks and computers from a new school programme share them with children in the neighborhood who are not part of the programme. Training through a farmer field school induces widespread adoption of integrated pest management among beneficiary farmers in a target community. This leads to a general reduction in the pest population reducing crop damage to non-beneficiaries.

OPTIONS Externalities General equilibrium effects Interactions Behavioral effects

ANSWERS 1. d, 2. b, 3. c, 4 a

//Slide 13:// Discussion Forum

Do you think there are spillover effects in programmes you are currently or previously worked on? What types of spillovers? Has any attempt been made to capture the importance of these spillover effects?


 * IV. Indicators** **and sources of information**

//Slide 14://

The end result of discussions with stakeholders on the theory of change, the evaluation questions, intended and unintended impacts and spillover effects should be a clear idea of the indicators that should be used to assess the effects of the programme.

These indicators: Should be noted in the programme logic or results chain that is created as part of the programme Must have a //**vertical logic**// in that the resources and activities that are part of the programmes planned work should lead to the programmes outputs, which should then lead to the programmes short-term and intermediate outcomes and ultimately to the long-term outcomes and impacts Must have a clear definition and/or formula for calculation, including the unit at which they will be measured Must be **SMART:**
 * S**pecific,
 * M**easurable,
 * A**chievable,
 * R**ealistic,
 * T**ime-bound

//Slide 15://

The final key piece of the puzzle for this stage of designing an impact assessment for a programme is to identify the source of information for each of the indicators in the programme logic or results chain. The source of information depends on whether the indicator is intended to capture the factual or the counterfactual Factual information is usually collected as part of a monitoring system and refers to facts about the programme and includes: Resources/inputs and activities that are part of the planned work of the programme Outputs and some short-term or intermediate outcomes that can be obtained as part of the programme monitoring The counterfactual refers to the need for attribution in assessing the programme impact and the requirement to collect information on what would have happened in the absence of the programme. It usually includes: Short-term or intermediate outcomes Long-term outcomes and impact The data collection then depends on whether the indicator is an attempt to determine facts about the program or if it is an attempt to attribute an effect to the programme.

//Slide 16://



//Slide 17://

Next week (Week 2) of this course will focus on the creation of a counterfactual and the approaches to assessing impact.

But before proceeding to discuss those approaches a clear sense of the source of information for indicators linked to impact need to be identified.

Where will the necessary information be obtained? What are the specific approaches to be used? Some possibilities include: Individual or household level surveys Anthropometric measurement Community or market level surveys Food or crop diaries Data collection in health centers, schools, etc. Secondary data from existing or planned surveys or censuses, existing government data collection, etc.

For every indicator, the source of data must be identified.

//Slide 18:// . QUESTION Which of the following is not easily collected in a individual or household level survey?

Use of agricultural inputs such as pesticides, fertiliser, labour, etc. Measurement of child height and weight for use in calculating height-for-weight, weight-for-age and height-for-age measurements Expenditures on food, health, schooling, clothing, etc. Labour time use and earning for all members of the household Prices received for goods produced in a microenterprise Adoption of a new agricultural technology

ANSWER There are challenges in each of these but they can be measured through households surveys with the exception of b). Since precision is very important, this is best measured using anthropometric measurement of children directly in the field.

//Slide 19:// Discussion Forum

Many development programmes have a particular emphasis on improving the well-being of poor women. How should gender issues influence the manner in which your indicators of impact are decided upon, chosen and collected?

//Slide 20://

To summarize: 1.Setting up an impact evaluation requires thinking through the theory of change and having a clear understanding of how programme activities will unltimately lead to an impact. 2. This theory of change helps in identifying key evaluation questions, including both intended and unintended impacts of the programme. 3. In assessing the overall impact of the programme, the possibility that there may be spiillover (indirect) effects needs to be considered. These my be as important as direct effects and have implications for the design of the impact assessment. 4. By carefully thinking through the theory of change, the evaluation questions and the possibility of spillover effects, it is easier to come up with SMART indicators to assess the effects of a programme on outcomes and impact measures.

//Slide 21://

Angelucci, M. and V. Di Maro. 2010. “Project Evaluation and Spillover Effects.” Impact Evaluation Guidelines, Strategy Development Division, Technical Notes No. IDB-TN-136. Inter-American Development Bank, Washington, DC. Gertler Paul, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch (2011) [|Impact Evaluation in Practice]. Washington DC: The World Bank. Stecklov G, Winters P, Todd J, Regalía F (2007) Unintended effects of poverty programmes on childbearing in less developed countries: Experimental evidence from Latin America. Population Studies 61(2):125-140