← MethodsSCREEN READING EDITIONDownload the PDF

ED-01 — The counterfactual is the missing column.

i

Summary

A rise after a program shows that something changed. To estimate how much of that change the program caused, you need a credible comparison.

ii

Full text equivalent

Example. The same 100 people answer a survey about their confidence when they start the program and again six months later.

The people, places, organizations and figures in this document's examples are fictional.

The measure is how confident people feel about finding their way around local services.

Change over time: 67% - 42% = +25 percentage points. This is the change we observed.

Effect of the program: 67% - ? = unknown until it is credibly estimated. This is the comparison we still need.

The starting point does not tell you what the future would have been without the program. You need to estimate what would have happened to these people at six months if they had not taken part.

Use it when: Someone describes a before-and-after improvement as impact.

Ask next: What evidence estimates what would have happened without the program?

Boundary: These results describe only the respondents who answered both times. A comparison is not credible just because it exists.

iii

Visual description

Three columns show confidence among the same 100 respondents. At intake, 42 of 100 are confident, and six months later, 67 of 100 are confident. The third column stands for what their confidence would have been at six months without the program, which was not observed. The observed increase is 25 percentage points. The effect of the program is 67 percent minus a credible estimate of the third column, and it remains unknown here.

iv

Evidence receipt

Gertler et al. · Impact Evaluation in Practice (2016), pp. 47-55

Chapter 3 supports the difference between observed change and the counterfactual, which is what would have happened without the program.