← MethodsSCREEN READING EDITIONDownload the PDF

Why the Right Grade Depends on the Decision

Assess fitness for use rather than a universal method ranking

Use when:you need to choose evidence that fits the question and what is at stake.

1

The short answer

The right grade for evidence depends on the decision it serves, and no single ranking of methods works for every case. Judge whether the evidence fits the actual question and how it will be used. A well-checked description can answer a question about delivery. An estimate from a study that offered the program at random can answer a question about effect while leaving open whether the result would transfer elsewhere, whether it is fair and whether the program is practical.

2

Match the job to the support

QuestionWhat support could fit
How many people received a service?Reliable records with clear definitions, with duplicates removed and exclusions stated
What did people experience?Suitable qualitative or quantitative inquiry, with an open account of who was chosen
Did the intervention make a difference?A credible comparison for judging causes, or a careful explanation of how the change came about
Should we act here?Relevant findings, together with priorities, costs, context, uncertainty and who has authority

Descriptions, associations, experiments and quasi-experiments are not four fixed grades. Each rests on assumptions, and each can be done well or badly. A more technical analysis cannot fix the wrong outcome or a population that does not fit the question.

3

Worked case: changing opening hours

A service is considering a $5,000 pilot of new opening hours. A proposed $40,000 evaluation might estimate the effect on a later outcome. For the pilot itself, existing attendance records, contact with people who could not attend and an assessment of staffing may answer the relevant question, which is whether the change is practical, sooner.

The people, places, organizations and figures in this document's examples are fictional.

“The proposed pilot is based on checked demand and access information, with review conditions. We have not established its effect on longer-term outcomes.”

Calling the change cheap and easy to reverse is not enough, because transport, childcare, work and accessibility can make it costly for participants. Check the arrangements for reversing it and ask the people affected. An action taken for now may be justified without any claim that it caused success.

The $40,000 inquiry is not automatically a waste. It may help with repeating the program elsewhere, with accountability, with another institution or with an evaluation that is required. Say what those uses are, when they arise and how much work they involve before you accept or reject the inquiry.

4

Changed condition: permanent closure

Now suppose the same organization is considering a $500,000 restructuring that would close sites for good. Attendance counts alone leave out where people would go, what other options they have and who would lose access. Look again at the question and at what evidence it needs. A randomized study is not automatically the right choice, and credible earlier evidence may reduce the need for a new one. The higher stakes call for more careful judgment, and not for a method chosen in advance because of its label.

5

Practice: evaluate the actual warrant

Try: A matched comparison is labeled “strong evidence,” but it matches people on employment measured after they took part. Is the label enough?

Worked answer: No. Choosing comparison groups on something measured after participation can distort the comparison. Look at how people came to be in each group, how they differed before the program, how far the groups overlap, what unmeasured differences could explain the result and what question the outcome answers. If the evidence does not support a causal reading, report the association. If the effect matters, commission suitable help with the design.

If a credible evaluation supports a useful causal result in a relevant population, report it as a positive finding, with its scope and uncertainty. Do not reject it just because a method that is ranked higher in name could exist. In the other direction, a positive estimate from a design that does not suit the question is not made trustworthy by its label.

6

Your evidence-fit record

Complete:

What to recordYour entry
purpose
precise question
institutional authority
claim compared with action
population and alternative
method and the assumptions it relies on
quality and coverage
relevance here
uncertainty
harms from acting and from waiting
how new information would be used now and later
cost and feasibility
justified next step
review owner

Look at the choice again when the question, the population or what is at stake changes. The record should let others follow the reasoning, including a sound decision to use the information you have or to look for more.

Your entries are saved in this browser only.
7

Completed working record and your copy

Use the model to see the reasoning. Then fill in the final column for your own case. If a fact or permission you need is missing, write it down as unresolved. Do not copy the model's conclusion.

FieldCompleted modelYour case
PurposeChoose evidence for a pilot of new opening hours
Relevant recordA proposed $5,000 pilot and a $40,000 inquiry; questions about access and staffing
Supported conclusion and limitMethod labels and budget alone do not show whether evidence fits
Next useful checkCheck uses now and later, harms to participants and whether the plan is practical
Owner and authorityThe service lead prepares a briefing; the person with authority to decide weighs the options
Cost, timing and permissionsCheck the local cost, timing and permissions before you promise the next check
Review triggerRevisit when coverage, definitions, relevant conditions or the evidence change

Explain why your conclusion follows, and say what change in the conditions would alter it. The worked practice above gives a contrasting task.

Related: How Much Evidence Is Enough?, for how much evidence an action needs; Quasi-Experimental Evidence, for judging comparison designs.

Your entries are saved in this browser only.
8

Further reading

  1. 1
    HM Treasury, Magenta Book. Sections 1.3 to 1.6 cover learning, accountability, use and planning.
  2. 2
    Impact Evaluation in Practice, second edition (2016). Chapters 3 to 4, 6 to 8 and the discussion of attrition cover counterfactuals and the assumptions behind designs.
  3. 3
    CDC, Program Evaluation Framework (2024). Steps 4 to 5 cover questions, measures, interpretation and recommendations that can be carried out.