impact evaluation matters: how long do i have to wait to know if my program works?

King and Behrman have a paper on Timing and Duration of Exposure in Evaluations of Social Programs. The paper gives a useful (if not adrenaline-fueled – this is a review, after all) discussion of a host of issues to consider when deciding when to look for results of social programs. [For example, what if the program ends up rolling out at different times in different places? etc] They then give lots of examples of papers that have dealt with the issues (and how they’ve done it). Instructive stuff.

Summary: Impact evaluations aim to measure the outcomes that can be attributed to a specific policy or intervention. Although there have been excellent reviews of the different methods that an evaluator can choose in order to estimate impact, there has not been sufficient attention given to questions related to timing: How long after a program has begun should one wait before evaluating it? How long should treatment groups be exposed to a program before they can be expected to benefit from it? Are there important time patterns in a program’s impact? Many impact evaluations assume that interventions occur at specified launch dates and produce equal and constant changes in conditions among eligible beneficiary groups; but there are many reasons why this generally is not the case. This paper examines the evaluation issues related to timing and discusses the sources of variation in the duration of exposure within programs and their implications for impact estimates. It reviews the evidence from careful evaluations of programs (with a focus on developing countries) on the ways that duration affects impacts.