Examine trust, exposure, burden, and refusal before treating missing responses as a technical problem
A follow-up survey reaches only half of participants. The response plan calls for more reminders. Staff add text messages, phone calls, and another incentive. Response improves slightly, but nobody asks why people did not want or feel able to answer in the first place.
Missing responses can be a logistical problem. They can also be information about trust, burden, accessibility, exposure, and the relationship between the program and the people it measures.
The short answer
Investigate the conditions around refusal and disappearance, not only the rate. Ask what people were told, what they believed could happen to their answers, whether participation felt optional, how burdensome the request was, and who was least able to respond. Check contact, accessibility, trust, burden and protection conditions and narrow the claim before intensifying collection. The cause of missingness remains unknown unless investigated.
The people missing from the record may be telling you something about the measurement system, even when they leave no completed form.
Why people may not appear
Contact barriers
Phone numbers change, mail is unstable, work schedules conflict, transportation is limited, internet access varies, and follow-up occurs at inaccessible times or in inaccessible formats.
Burden
People are asked the same questions repeatedly, surveys are too long, the emotional cost is high, or the value of responding is unclear.
Exposure
Questions concern housing, income, health, family, immigration, justice involvement, or other circumstances that can feel risky to disclose.
Power
Participants may believe answers affect eligibility, staff treatment, benefits, legal status, or future access. A consent statement does not erase that belief.
Experience of the service
People who left early, felt poorly served, or were harmed may be least willing to remain in contact. Their absence can make the visible result look more favorable.
Success or changed priorities
Some people no longer need the program, are busy with work or family, or see no reason to continue contact. Do not assume every missing outcome is poor.
The key question is whether the reasons for missingness could be related to the outcome or experience being reported.
A worked example
A peer-support program uses a wellbeing survey at intake, every month, and exit. People who stop attending are contacted up to six times. Staff notice that participants with unstable housing and people who reported negative early experiences are less likely to complete exit surveys.
The people, places, organizations and figures in this document's examples are fictional.
The program shortens the monthly survey, stops asking two sensitive questions that have no current use, explains the actual access consequences of answering or refusing, and removes a linkage to service eligibility where authorized, offers phone, text, paper, and supported completion, and asks a separate optional question about why someone leaves. Follow-up is conducted by a person outside the direct service relationship where feasible.
The program also changes its report. It shows outcome coverage and the known differences between respondents and nonrespondents. The visible wellbeing result becomes less sweeping but more trustworthy.
Review the measurement relationship
Ask participants and frontline staff:
- What do people think the information is for?
- Do they believe saying no is safe?
- Could an answer affect service, funding, immigration, custody, supervision, or other consequences?
- Which questions feel repetitive, intrusive, confusing, or irrelevant?
- Does the timing make sense in the participant's life?
- Are language, disability, literacy, technology, and compensation addressed?
- Who asks the questions, and what relationship do they hold?
- Do participants ever receive or benefit from what is learned?
Do not ask only the people who continue responding. Seek routes to hear from people who declined or left without making renewed participation feel compulsory.
Practical safeguards
Explain the use plainly. Say who will see the information, what decisions it affects, and what it will not be used for.
Make refusal real. Separate measurement from service eligibility where possible and train staff not to pressure participation.
Reduce the request. Remove questions without a real decision, obligation, or learning use.
Offer accessible routes. Use appropriate language, format, timing, technology, and support.
Separate roles when useful. People may answer differently when the person measuring does not control their service.
Return value. Share what was learned and what changed because people gave their time.
Track reasons carefully. Record refusal and loss to follow-up without turning the reason into another compulsory disclosure.
What you can say
- "Outcomes were available for this proportion of participants, with lower coverage in these groups."
- "Some people reported that the questions felt repetitive or exposed."
- "We reduced collection and clarified that responses do not determine access."
- "The result describes respondents and may not represent people who left earlier."
What you cannot say yet
- "Nonrespondents are probably doing worse," without support.
- "Participants consented freely," solely because a form was signed.
- "More reminders solved missing data," if the underlying trust problem remains.
- "The survey is low burden," based only on completion time.
What to do next
Minimum useful step: Report response counts and rewrite claims to name respondents rather than all participants.
Stronger step: Examine response patterns and conduct a trust-and-burden review with people affected, including those who stopped participating where appropriate.
Prospective step: Redesign the collection purpose, consent, roles, timing, access, and return of findings. Set a review point for whether the new approach improves both coverage and candor.
A 15-minute team exercise
Choose one survey, interview, or follow-up process.
- 1Write what decision or obligation each requested item serves.
- 2Imagine the process from the participant's position.
- 3Mark where refusal may not feel safe.
- 4List groups most likely to face access or exposure barriers.
- 5Remove one low-value question or contact.
- 6Improve one explanation, format, or boundary.
- 7Decide how participants will hear what changed because of the information.
Do not end with a plan to contact people more often unless the team has first addressed why contact may be unwelcome or difficult.
A note on surveillance
Measurement drifts toward surveillance when information is collected because it can be, follows people beyond a proportionate use, or is used for consequences they did not reasonably expect. Programs operating in child welfare, justice, immigration, health, housing, or benefits systems should be especially careful.
Ask whether the system would feel fair if the person being measured could see, correct, and question the record. Trust is not a soft addition to data quality. It affects who appears, what they say, and whether the evidence can be believed.
When to bring in help
Outside support may be needed when collection involves sensitive information, legal or safeguarding consequences, strong power imbalances, persistent differential missingness, or evidence of harm. Participant-led expertise should sit beside methodological and privacy expertise.
The useful output is a measurement arrangement people can recognize as proportionate, safe enough for candor, and connected to a real use.
Questions to carry forward
- What do people believe will happen to their answers?
- Does refusal feel safe in practice?
- Who is least likely to remain visible, and why?
- Which collection burden can be removed before asking people for more?
Further reading
- 1American Evaluation Association, Guiding Principles. Principles A2–A5, C1–C7, D1–D4 and E1–E4: inquiry, integrity, people and equity.
- 2GAO, Assessing Data Reliability, GAO-20-283G (2019). The official overview: data accuracy, completeness and applicability to purpose.
- 3Spencer, Ritchie, Lewis and Dillon, Quality in Qualitative Evaluation (2003; Treasury republication 2012). Appraisal questions 1–2 and 12–14: evidence, contrary accounts and traceable interpretation.