← MethodsSCREEN READING EDITIONDownload the PDF

The Trial Tested the Offer

A randomized offer can answer a useful policy question even when many people never take it up. The difficulty begins when the report turns that answer into a claim about receiving the service.

A county funds financial coaching and wants to know whether it reduces household arrears. Eligible households receive invitations, appointments are offered, and some people complete the sessions. Others never open the message, cannot attend during working hours, or leave before the course ends.

At the end of the trial, the county has several possible questions. Did offering coaching help the eligible population? Did completing coaching help the people who received it? Would a different enrollment process produce a larger benefit?

Those questions concern different interventions and populations. Randomization does not make their answers interchangeable.

Start with the comparison the trial created

Consider an example. The county randomly assigns 2,000 eligible households in equal groups to a new coaching offer or existing practice. In the offer group, 46 percent complete a defined coaching course. In the comparison group, 10 percent complete the same course through an existing access route. Completion has the same definition in both groups.

The people, places, organizations and figures in this document's examples are fictional.

One year later, 28.2 percent of the offer group and 30.0 percent of the comparison group are in arrears. Assume that the outcome is measured consistently for all 2,000 households.

The assignment-effect estimate is a reduction of 1.8 percentage points: 28.2 minus 30.0. This is the intention-to-treat effect, or ITT. It compares households as originally assigned, regardless of whether they completed coaching.

That is often the effect a government needs. It can fund an offer, arrange outreach and provide a service. It usually cannot guarantee that every eligible person participates. The World Bank's Impact Evaluation in Practice makes this policy argument in its discussion of randomized offering: the effect of an offer is useful precisely because participation remains voluntary.

Low take-up is therefore part of the result. Confusing invitations, inconvenient hours or difficult enrollment may limit what the current offer achieves. Calling the ITT a diluted version of the “true effect” can hide the implementation decision the county actually faces.

The 1.8-point figure is a point estimate, not a complete decision case. A real report would also need uncertainty estimates, costs, possible harms and a judgment about the practical size of the benefit.

Receiving the service is a different causal question

Comparing completers with noncompleters usually breaks the protection supplied by random assignment. People who complete may have different schedules, motivation, support or financial risks. These differences can influence arrears even without coaching.

Under specific assumptions, randomized assignment can instead be used as an instrumental variable to estimate an effect of receipt. Here, the offer increased course completion by 36 percentage points: 46 minus 10. Dividing the outcome difference by the completion difference gives:

−0.018 ÷ 0.36 = −0.05: an estimated 5-percentage-point reduction in arrears.

The arithmetic is straightforward. The interpretation carries the difficulty.

In the Angrist–Imbens–Rubin framework, this is a local average treatment effect: an effect for people whose participation changes because of the randomized offer. These people are often called compliers. It is not automatically an effect for everyone who participated or everyone eligible.

The assumptions include:

These assumptions supplement the randomized assignment; they do not follow from the division itself.

The World Bank handbook also makes a less familiar point: the complier group is not a list the county can simply pull from its attendance records. Among completers who received the offer, some would have completed anyway and some completed because of it. The records do not reveal both possibilities for the same household. The causal estimate can describe a group that cannot be identified person by person.

A different invitation, appointment system or incentive may change who is induced to participate. Its effect of receipt may therefore concern a different population.

Keep delivery and follow-up visible

A useful report should let the reader distinguish what happened at each stage: eligibility, assignment, contact, enrollment, completion and outcome measurement. Each loss has a different meaning.

A household that never completes coaching still belongs in its assigned group for an ITT analysis. A household whose outcome cannot be observed creates a missing-data problem. Keeping the original group label does not, by itself, remove bias from selective follow-up.

The CONSORT 2025 statement calls for participant-flow reporting and descriptions of the intervention as actually delivered, including adherence and fidelity. For a policy team, that reporting discipline helps separate three decisions:

FindingDecision it can inform
Effect of the existing offer on those assignedWhether to continue or expand that offer under comparable conditions
Where people lose access or fail to receive the intended serviceWhich part of delivery needs investigation or redesign
An effect of receipt identified under additional assumptionsWhat the service may achieve for the population induced to participate

Delivery records can locate a problem without proving the effect of fixing it. If many households drop out while scheduling, the county has a reason to examine scheduling. It has not yet established that removing that barrier will reproduce the trial's receipt-effect estimate across all households.

Put the decision into the claim

Before analysis, complete this sentence:

We need to estimate the effect of [assignment, access, receipt or completion] for [which population], under [which delivery conditions], because the decision is [specific decision].

For the county, a defensible summary begins: “Assignment to the current coaching offer reduced the estimated arrears rate by 1.8 percentage points over one year.” The report should then supply the uncertainty, participation pattern and limits needed to interpret that estimate.

If the next decision concerns enrollment redesign, commission the evidence for that change. If it concerns the effect of receiving coaching, state the identified population and the assumptions required. A trial can support several useful analyses, but the reader needs to know which question each one answers.

Before accepting the headline, ask: What exactly did the trial make different—and is that the thing the headline says worked?

Related: Before-and-after data, for why observed improvement needs a credible comparison; The Bureaucracy Is Part of the Treatment, for how delivery arrangements change the intervention people receive.

Further reading

  1. 1
    Gertler et al., Impact Evaluation in Practice, first edition (2011), chapter 4, printed pp. 65–69 — randomized offering, imperfect compliance and the population identified by the receipt-effect calculation.
  2. 2
    Angrist, Imbens and Rubin, “Identification of Causal Effects Using Instrumental Variables” (1996) — the assumptions and interpretation of instrumental-variable estimates.
  3. 3
    Hopewell et al., CONSORT 2025 statement, items 22a and 24a — participant flow and actual intervention delivery.
  4. 4
    Impact Evaluation in Practice, second edition (2016) — an expanded treatment; chapters 4 and 5 cover randomized assignment and instrumental variables.