← MethodsSCREEN READING EDITIONDownload the PDF

Find the Comparison Already Inside the Work

i

How waitlists, thresholds, capacity limits, and phased rollout can create useful learning opportunities

For:program designers, public agencies, nonprofit leaders, funders, evaluators, and teams planning access or rollout
Use when:random assignment is not planned but the way people or sites receive access may create a credible contrast
Includes:an access-pathway review, four comparison opportunities, assumption checks, record checklist, and a 15-minute team exercise
Brief paths:start new work; expand; compare programs; clarify causal uncertainty; review program or administrative data

Programs often decide who receives access before anyone thinks about evaluation. A waitlist forms. Capacity is limited. Sites begin at different times. Eligibility follows a score or age threshold. These operating choices may create useful comparisons.

They do not create credible evidence automatically. The reason one person or site receives access before another is central to what can be learned.

ii

The short answer

Map the actual access process, not only the written rule. Look for moments where similar people or sites receive different access because of timing, capacity, thresholds, or allocation. Preserve the rule, overrides, dates, and baseline information needed to assess the comparison. Treat each opportunity as a design to examine, not as proof of an effect.

The comparison is not simply the group that did not receive the program. It is the best available account of what might have happened otherwise.

iii

Begin with the access pathway

Write the full sequence:

  1. 1
    Who could be eligible?
  2. 2
    Who becomes known to the program?
  3. 3
    Who is assessed or referred?
  4. 4
    Who is offered access?
  5. 5
    Who accepts and starts?
  6. 6
    Who receives a meaningful amount of service?

At each point, record who makes the decision, which rule applies, what discretion exists, and which information is saved. The operational pathway often differs from the policy manual.

iv

Four comparison opportunities

Waitlists and limited capacity

When demand exceeds capacity, people who enter and people who wait may appear comparable. Ask how queue position is determined. If staff prioritize urgency, readiness, geography, or likelihood of success, the groups may differ before service begins.

Random selection among equally eligible applicants can create a strong comparison when it is fair and operationally acceptable. A first-come queue may also be informative, but application speed can reflect internet access, schedule, language, or institutional familiarity.

Phased rollout

Different sites or communities begin at different times. The later group can sometimes show what happened during the same period without the program. Ask why early sites were selected. Readiness, leadership quality, political support, or greater need may also affect outcomes.

Record the rollout decision before results and track other changes affecting early and later sites.

Eligibility thresholds

Access may begin above or below a score, age, income, or risk threshold. People close to either side can sometimes support a credible comparison because they are similar except for the rule.

The opportunity weakens when scores are manipulated, discretion is common, the threshold changes, or other services also depend on the same cutoff.

Assignment rules or lotteries

A formal allocation process may distribute places, staff, or sites. Preserve the list of eligible units, the procedure, random seed or draw record where relevant, original group labels separately from actual offers/receipt, refusals, crossover, and any overrides. Record each override's stage, reason, date and authority.

A fair allocation process can serve both operations and learning, but evaluation value should not be used to justify withholding a service people are entitled to receive.

v

A worked example

A legal-support program can serve 300 of 480 eligible tenants before their hearings. Staff currently choose cases based on urgency and the completeness of intake documents. Comparing served and unserved tenants would be misleading because those choices are related to case outcomes.

The people, places, organizations and figures in this document's examples are fictional.

The program reviews its obligations and determines that 220 high-urgency cases must continue to receive priority. Among the remaining eligible cases, 80 places can be allocated by a transparent lottery. The program records the eligible list, draw, offers, refusals, hearing dates, and outcomes.

Of the 480 eligible tenants, 220 receive priority and 260 remain lottery-eligible; 80 are assigned an offer and 180 are not. In the baseline case offers follow that draw. The primary randomized comparison concerns original assignment among those 260, retaining refusals and crossover rather than comparing only actual service recipients. Valid, comparable outcomes and appropriate follow-up/analysis are still needed. The resulting comparison answers a bounded question about the lottery-eligible group. It does not describe the highest-urgency cases or prove the program works for every tenant. The design respects operational priority while creating a credible learning opportunity within the remaining capacity.

vi

Assumptions to examine

For any naturally occurring comparison, ask:

No design is saved by a clever label if these assumptions do not hold.

vii

Records worth preserving

Keep only what the design and decision need, but preserve it contemporaneously:

Retrospective staff memory can help explain a process. Check it against dated records and contrary accounts; both records and recollections can be inaccurate. Do not create a supposedly contemporaneous allocation rule after seeing the result.

viii

What you can say

ix

What you cannot say yet

x

What to do next

Minimum useful step: Draw the access pathway and identify where different access occurs.

Stronger step: Examine the allocation rule, discretion, baseline similarity, alternative services, outcome measurement, and missing follow-up before selecting a design.

Prospective step: Where lawful and fair, adjust rollout or allocation to preserve a stronger learning opportunity. Agree the analysis and use before outcomes are visible.

xi

A 15-minute team exercise

Choose one program with limited capacity, phased rollout, or a formal eligibility rule.

  1. 1
    Draw the pathway from eligibility to receipt of service.
  2. 2
    Mark every point where staff or participants make a choice.
  3. 3
    Circle the point that creates different access.
  4. 4
    Ask why units fall on either side of that point.
  5. 5
    List the records currently preserved.
  6. 6
    Name the most important assumption a comparison would require.
  7. 7
    Decide whether the opportunity deserves a design review before operations change.

If the pathway cannot be reconstructed, start recording it now. Do not retrofit certainty onto earlier allocation.

xii

A note on fairness and rights

Evidence value does not override entitlement, urgency, informed consent, or a duty to provide care. Do not manufacture scarcity or delay access solely to create a comparison when a fairer route is available.

When scarcity already exists, a transparent rule may be fairer than hidden discretion and more informative at the same time. The ethical and operational decision comes first. The evidence design must fit inside it.

1

When to bring in help

Bring in design support before changing access, rollout, or record systems. This is especially important when eligibility affects rights, benefits, liberty, health, or essential services. Legal, ethical, operational, and community expertise may be needed alongside evaluation expertise.

A useful review should state the population the comparison can represent, the assumptions it depends on, the records needed, and the claims it could never support.

14

Questions to carry forward

15

Further reading

  1. 1
    American Evaluation Association, Guiding Principles. Principles A2–A5, C1–C7, D1–D4 and E1–E4: inquiry, integrity, people and equity.
  2. 2
    GAO, Assessing Data Reliability, GAO-20-283G (2019). The official overview: data accuracy, completeness and applicability to purpose.
  3. 3
    Spencer, Ritchie, Lewis and Dillon, Quality in Qualitative Evaluation (2003; Treasury republication 2012). Appraisal questions 1–2 and 12–14: evidence, contrary accounts and traceable interpretation.
  4. 4
    Gertler and colleagues, World Bank/IDB, Impact Evaluation in Practice, second edition (2016). Chapters 3–4, 6–8 and the discussion of attrition: counterfactuals, assignment, design assumptions and missing follow-up. Chapter 5, printed pp. 91–94 (PDF pp. 121–124): comparing the original randomized groups despite noncompliance.