How waitlists, thresholds, capacity limits, and phased rollout can create useful learning opportunities
Programs often decide who receives access before anyone thinks about evaluation. A waitlist forms. Capacity is limited. Sites begin at different times. Eligibility follows a score or age threshold. These operating choices may create useful comparisons.
They do not create credible evidence automatically. The reason one person or site receives access before another is central to what can be learned.
The short answer
Map the actual access process, not only the written rule. Look for moments where similar people or sites receive different access because of timing, capacity, thresholds, or allocation. Preserve the rule, overrides, dates, and baseline information needed to assess the comparison. Treat each opportunity as a design to examine, not as proof of an effect.
The comparison is not simply the group that did not receive the program. It is the best available account of what might have happened otherwise.
Begin with the access pathway
Write the full sequence:
- 1Who could be eligible?
- 2Who becomes known to the program?
- 3Who is assessed or referred?
- 4Who is offered access?
- 5Who accepts and starts?
- 6Who receives a meaningful amount of service?
At each point, record who makes the decision, which rule applies, what discretion exists, and which information is saved. The operational pathway often differs from the policy manual.
Four comparison opportunities
Waitlists and limited capacity
When demand exceeds capacity, people who enter and people who wait may appear comparable. Ask how queue position is determined. If staff prioritize urgency, readiness, geography, or likelihood of success, the groups may differ before service begins.
Random selection among equally eligible applicants can create a strong comparison when it is fair and operationally acceptable. A first-come queue may also be informative, but application speed can reflect internet access, schedule, language, or institutional familiarity.
Phased rollout
Different sites or communities begin at different times. The later group can sometimes show what happened during the same period without the program. Ask why early sites were selected. Readiness, leadership quality, political support, or greater need may also affect outcomes.
Record the rollout decision before results and track other changes affecting early and later sites.
Eligibility thresholds
Access may begin above or below a score, age, income, or risk threshold. People close to either side can sometimes support a credible comparison because they are similar except for the rule.
The opportunity weakens when scores are manipulated, discretion is common, the threshold changes, or other services also depend on the same cutoff.
Assignment rules or lotteries
A formal allocation process may distribute places, staff, or sites. Preserve the list of eligible units, the procedure, random seed or draw record where relevant, original group labels separately from actual offers/receipt, refusals, crossover, and any overrides. Record each override's stage, reason, date and authority.
A fair allocation process can serve both operations and learning, but evaluation value should not be used to justify withholding a service people are entitled to receive.
A worked example
A legal-support program can serve 300 of 480 eligible tenants before their hearings. Staff currently choose cases based on urgency and the completeness of intake documents. Comparing served and unserved tenants would be misleading because those choices are related to case outcomes.
The people, places, organizations and figures in this document's examples are fictional.
The program reviews its obligations and determines that 220 high-urgency cases must continue to receive priority. Among the remaining eligible cases, 80 places can be allocated by a transparent lottery. The program records the eligible list, draw, offers, refusals, hearing dates, and outcomes.
Of the 480 eligible tenants, 220 receive priority and 260 remain lottery-eligible; 80 are assigned an offer and 180 are not. In the baseline case offers follow that draw. The primary randomized comparison concerns original assignment among those 260, retaining refusals and crossover rather than comparing only actual service recipients. Valid, comparable outcomes and appropriate follow-up/analysis are still needed. The resulting comparison answers a bounded question about the lottery-eligible group. It does not describe the highest-urgency cases or prove the program works for every tenant. The design respects operational priority while creating a credible learning opportunity within the remaining capacity.
Assumptions to examine
For any naturally occurring comparison, ask:
- What makes the proposed comparison credible beyond similarity on recorded characteristics?
- Could people influence which group they entered?
- Did staff override the rule, and why?
- Did people in one group receive similar services elsewhere?
- Could the program affect the comparison group indirectly?
- Were outcomes measured the same way for both groups?
- Who disappeared from follow-up in each group?
- Did another policy or event affect one group differently?
No design is saved by a clever label if these assumptions do not hold.
Records worth preserving
Keep only what the design and decision need, but preserve it contemporaneously:
- Eligibility rules and changes.
- Referral and application dates.
- Baseline characteristics relevant to access and outcome.
- Queue, threshold, rollout, or allocation records.
- Offers, acceptance, refusal, crossover, and overrides.
- Delivery dates and amount of service received.
- Outcome source, period, and coverage.
- Other services or major events likely to affect the comparison.
Retrospective staff memory can help explain a process. Check it against dated records and contrary accounts; both records and recollections can be inaccurate. Do not create a supposedly contemporaneous allocation rule after seeing the result.
What you can say
- "The phased rollout creates a possible comparison that depends on why sites started at different times."
- "Among people equally eligible for the remaining places, access was allocated by lottery."
- "The threshold design applies to people close to the cutoff, not the entire population."
- "We preserved overrides and crossover so the planned comparison can be assessed honestly."
What you cannot say yet
- "People on the waitlist are a control group," without understanding the queue.
- "Early and late sites are comparable," when readiness determined rollout.
- "The threshold proves causation," without checking manipulation, other cutoffs, and measurement.
- "Nonparticipants show what would have happened," when participation was voluntary and self-selection is unexamined.
What to do next
Minimum useful step: Draw the access pathway and identify where different access occurs.
Stronger step: Examine the allocation rule, discretion, baseline similarity, alternative services, outcome measurement, and missing follow-up before selecting a design.
Prospective step: Where lawful and fair, adjust rollout or allocation to preserve a stronger learning opportunity. Agree the analysis and use before outcomes are visible.
A 15-minute team exercise
Choose one program with limited capacity, phased rollout, or a formal eligibility rule.
- 1Draw the pathway from eligibility to receipt of service.
- 2Mark every point where staff or participants make a choice.
- 3Circle the point that creates different access.
- 4Ask why units fall on either side of that point.
- 5List the records currently preserved.
- 6Name the most important assumption a comparison would require.
- 7Decide whether the opportunity deserves a design review before operations change.
If the pathway cannot be reconstructed, start recording it now. Do not retrofit certainty onto earlier allocation.
A note on fairness and rights
Evidence value does not override entitlement, urgency, informed consent, or a duty to provide care. Do not manufacture scarcity or delay access solely to create a comparison when a fairer route is available.
When scarcity already exists, a transparent rule may be fairer than hidden discretion and more informative at the same time. The ethical and operational decision comes first. The evidence design must fit inside it.
When to bring in help
Bring in design support before changing access, rollout, or record systems. This is especially important when eligibility affects rights, benefits, liberty, health, or essential services. Legal, ethical, operational, and community expertise may be needed alongside evaluation expertise.
A useful review should state the population the comparison can represent, the assumptions it depends on, the records needed, and the claims it could never support.
Questions to carry forward
- Why does one person or site receive access before another?
- What discretion or self-selection enters the process?
- What records must be preserved before the result exists?
- Is the comparison fair, lawful, and useful for the actual decision?
Further reading
- 1American Evaluation Association, Guiding Principles. Principles A2–A5, C1–C7, D1–D4 and E1–E4: inquiry, integrity, people and equity.
- 2GAO, Assessing Data Reliability, GAO-20-283G (2019). The official overview: data accuracy, completeness and applicability to purpose.
- 3Spencer, Ritchie, Lewis and Dillon, Quality in Qualitative Evaluation (2003; Treasury republication 2012). Appraisal questions 1–2 and 12–14: evidence, contrary accounts and traceable interpretation.
- 4Gertler and colleagues, World Bank/IDB, Impact Evaluation in Practice, second edition (2016). Chapters 3–4, 6–8 and the discussion of attrition: counterfactuals, assignment, design assumptions and missing follow-up. Chapter 5, printed pp. 91–94 (PDF pp. 121–124): comparing the original randomized groups despite noncompliance.