The short answer
People can drop out of view before they enroll, while the service is being delivered, or at follow-up. Each kind of absence affects a different question. Define the population first, keep unknown outcomes recorded as unknown, and look into how leaving people out, or missing their data, might change the account.
Map five stages
- 1Eligible people who were never reached. This is a question about access.
- 2People who decline or cannot enroll. This is a question about entry.
- 3People who enroll and then leave early. This is a question about participation.
- 4People who cannot be reached at follow-up. This is a question about outcome coverage.
- 5People who were reached, but whose answer on the relevant item is missing or unusable. This is a question about the item or the measurement.
These stages can overlap, so do not add them up as if they were separate groups. A person who leaves early may also miss follow-up. People who were never served do not automatically belong in the denominator for outcomes of the enrolled cohort, although they matter for questions about reach and equity.
Worked case: housing
A prevention service enrolls 200 families, and all of them are due for follow-up at six months. The team gets a valid housing status for 120 families. Of these, 98 kept their housing and 22 did not. The outcomes for the other 80 are unknown.
The people, places, organizations and figures in this document's examples are fictional.
- Observed retention: 98 / 120 = 81.7%, or about 82%.
- Coverage: 120 / 200 = 60%.
- Share known to have kept housing: 98 / 200 = 49%.
“Among 120 families with six-month outcomes, 98 retained housing (81.7%). This covers 60% of the 200-family cohort; the other 80 outcomes are unknown.”
Assume each family is correctly classed as housed or not, using the same definition and date for all. Then retention for the whole cohort could be anywhere from 49% to 89%. If 40 of the missing families kept their housing, the rate would be 69%, and if all 80 did, it would be 89%. These figures are assumptions and limits. They are not an estimate or a confidence interval.
What absence might mean
A family might have moved into secure housing, lost access to a phone, wanted privacy, been struggling with housing or simply missed the contact. Silence does not tell you which way the error runs. Where it is appropriate, compare the characteristics recorded at the start. Do not conclude that because the groups look similar on what was measured, their unmeasured outcomes are also the same.
Statistical methods such as weighting or filling in missing values, or linking to administrative records, can help under the right conditions. They need clearly stated assumptions, suitable information to predict from, or checks on coverage, and they do not automatically bring back the full population. “Missing at random” is a statistical assumption that depends on what else is known about people. It does not mean that staff are unable to explain why a call was missed.
Changed condition and practice
Try: Safe follow-up reaches 20 previously missing families; ten retained housing. Recalculate.
Worked answer: Now 108 of 140 observed families kept their housing, which is 77.1%, with 70% coverage. Sixty outcomes are still unknown, which gives limits of 54% to 84% under the same assumptions. The lower observed rate is new information. It is not evidence that follow-up harmed anyone, and it does not show that the families still missing are like the ones just reached.
Your cohort-flow record
Record the following.
| What to record | Your entry |
|---|---|
| population and eligibility | |
| enrolled | |
| left early | |
| eligible at follow-up | |
| valid positive and negative | |
| unknown or unusable | |
| reasons known versus inferred | |
| safe contact options | |
| bounds or sensitivity assumptions | |
| supported wording | |
| next check and owner |
Explain why people were left out without calling the missing people uncooperative.
Respect people's contact preferences, their accessibility needs, their privacy and their choice about taking part. A lawful check of records may be appropriate, but contacting people again and again is not automatically in proportion to the need. The evaluation owner should look at the missing data before anyone approves an estimate for the whole cohort or a claim about cause. Looking into access and following up outcomes may both matter, but keep their denominators separate.
Completed working record and your copy
Use the model to see the reasoning. Then fill in the final column for your own case. If a fact or permission you need is missing, write it down as unresolved. Do not copy the model's conclusion.
| Field | Completed model | Your case |
|---|---|---|
| Purpose | Report housing retention when some outcomes are missing | |
| Relevant record | 200 families; 120 observed; 98 positive; 22 negative; 80 unknown | |
| Supported conclusion and limit | 81.7% among those observed; 60% coverage; whole-cohort limits of 49% to 89% | |
| Next useful check | Review safe follow-up, exclusions and assumptions about the missing data | |
| Owner and authority | The follow-up owner manages contact; the evaluator checks any claim about the whole group | |
| Cost, timing and permissions | Check the local cost, timing and permissions before you promise the next check | |
| Review trigger | Revisit when coverage, definitions, relevant conditions or the evidence change |
Explain why your conclusion follows, and say what change in the conditions would alter it.
Related: Can We Report Outcomes When Only Some Respond?, for reporting when only some people respond; Can We Claim Equity Progress?, for testing an equity claim.
Further reading
- 1AAPOR, Standard Definitions overview. It covers nonresponse and response rates.
- 2American Evaluation Association, Guiding Principles. See the principles on respect for people and on the common good and equity, including consent and power.