Summary
A rise after a program shows that something changed. To estimate how much of that change the program caused, you need a credible comparison.
Full text equivalent
Example. The same 100 people answer a survey about their confidence when they start the program and again six months later.
The people, places, organizations and figures in this document's examples are fictional.
The measure is how confident people feel about finding their way around local services.
- Observed: At intake, before the program, 42% are confident, which is 42 of 100 people.
- Observed: Six months later, with the program, 67% are confident, which is 67 of 100 people.
- Missing: Six months later, without the program, the figure is unknown. It would describe the same people, but this outcome cannot be observed.
Change over time: 67% - 42% = +25 percentage points. This is the change we observed.
Effect of the program: 67% - ? = unknown until it is credibly estimated. This is the comparison we still need.
The starting point does not tell you what the future would have been without the program. You need to estimate what would have happened to these people at six months if they had not taken part.
Use it when: Someone describes a before-and-after improvement as impact.
Ask next: What evidence estimates what would have happened without the program?
Boundary: These results describe only the respondents who answered both times. A comparison is not credible just because it exists.
Visual description
Three columns show confidence among the same 100 respondents. At intake, 42 of 100 are confident, and six months later, 67 of 100 are confident. The third column stands for what their confidence would have been at six months without the program, which was not observed. The observed increase is 25 percentage points. The effect of the program is 67 percent minus a credible estimate of the third column, and it remains unknown here.
Evidence receipt
Gertler et al. · Impact Evaluation in Practice (2016), pp. 47-55
Chapter 3 supports the difference between observed change and the counterfactual, which is what would have happened without the program.