← MethodsSCREEN READING EDITIONDownload the PDF

The Honesty Layer Rubric

How to review a claim against its evidence

An impact claim can outrun what was measured, even when the underlying records are useful. The Honesty Layer reviews the sentence and the evidence beneath it, asking whether the record and reasoning support what a reader would understand the sentence to mean. It also records a more accurate formulation when the original overreaches.

1The principle

A claim asks its evidence to do a particular job. “The program worked” may mean that services were delivered, that conditions improved, that the program had a causal role, or that expansion is justified. Each assertion needs its own support.

A modest claim, honestly made, is sound when the records, definitions and coverage support it. A stronger causal claim is also appropriate when the warrant supports it. The task is matching rather than climbing: neither a method label nor cautious wording makes a claim valid.

2Evidence jobs and causal warrants

2.1

Descriptive: what happened.

Counts, conditions and trajectories can support defined statements about observed people, records and periods. Comparable observations may describe change over time. Description alone does not attribute that change to the program. Check duplicates, definitions, missing records and who is absent before accepting even a modest count.

2.2

Associational: what moved together.

An association can describe a pattern, such as better outcomes among completers. Selection, confounding and timing matter. An association alone does not establish that completion produced the difference; nor does the fact that assignment was nonrandom make every possible causal inquiry invalid.

2.3

Causal: what difference or role the program made.

A credible comparison can estimate an effect under stated assumptions. Randomized allocation, cutoffs, instrumental variables and difference-in-differences require different checks. Randomization supports comparability in expectation at allocation; measurement, missing follow-up, crossover, spillovers and analysis remain consequential. For a lottery, the effect of an offer differs from an effect of actual participation.

Systematic qualitative causal inquiry can test mechanisms and rival explanations using timing, corroboration and discriminating evidence. It can support a bounded causal explanation without estimating an average effect. A contribution claim is a causal claim about a role alongside other causes, not a noncausal intermediate grade. Selected testimonials or a plausible theory of change alone do not establish that role.

2.4

Prediction, transfer and recommendation.

A credible effect in one study does not by itself establish what will happen elsewhere or which option should be chosen. Changed delivery, population, context, costs, harms, duties and alternatives need consideration. The evidence does not supply the decision-maker's authority.

These are different evidence jobs. Experimental and quasi-experimental are design families, not grades above description or qualitative inquiry.

3The three review outcomes

3.1

Supported within scope.

The inspected evidence supports the sentence, with its essential qualifications. Record people, period, intervention version, warrant and uncertainty. This is a review finding about the specified evidence, not a certificate of truth or institutional worth.

3.2

Revise.

The evidence supports a more bounded or different formulation. State that formulation and explain the change: population, construct, magnitude, horizon, causal wording or scope. Do not silently delete a favorable result or an important limit.

3.3

Unsupported as stated.

An essential part of the assertion lacks support, an outcome was never measured, a material threat remains unresolved, or the necessary records are unavailable. Name the gap. A distinct delivery or experience claim may still be supportable.

The distinction between revise and unsupported depends on what the evidence actually contains. It is not a score and need not force a single interpretation when reviewers disagree.

4What the rubric checks

  • Identification, when causation is asserted: what supports the causal interpretation, which assumptions remain, and which rival explanation would change it?
  • Measurement and coverage: what was measured, from which source, for whom, when, and with what missingness or definition changes?
  • Delivery and scope: which intervention was delivered, to which people, under which conditions? Imperfect receipt changes the question being estimated; it does not automatically invalidate an effect of assignment.
  • Claim-evidence match: do the outcome, denominator, magnitude, uncertainty, horizon and wording fit the evidence? Distinguish uncertainty about precision from bias and unmeasured outcomes.

Use the checks relevant to the assertion. A service count does not need an effect estimate; an effect estimate does not resolve an expansion decision. An important omission stays unresolved rather than being converted into a favorable score.

5Worked examples

5.1

1 · A causal claim on an association → Revise.

Claim: “Our program cut recidivism by 30%.” In checked records for the same outcome and follow-up period, 14 of 100 completers and 20 of 100 noncompleters have a recorded event. Rates are 14% and 20%: a six-percentage-point difference, or 30% lower relative to 20%. Completion was not assigned. Selection and other explanations remain unresolved. Revised claim: “Recorded recidivism was lower among completers than noncompleters in this cohort; the comparison does not establish the program's effect.” Population, period and record coverage must accompany a live report.

Recorded events in the same outcome and follow-up period

Completers

14 of 100

Noncompleters

20 of 100

Recorded event

The people, places, organizations and figures in this document's examples are fictional.

5.2

2 · A descriptive claim → Supported within scope.

Claim: “Enrollment rose 40% in the year after the redesign.” Checked, consistently defined records show 350 enrollments in the earlier year and 490 in the later one. The increase is 140/350, or 40%. Finding: the descriptive sentence fits those records. “The redesign drove the increase” adds a causal assertion the two counts alone do not support.

Enrollments
Earlier year
350
Later year
490
5.3

3 · A causal claim near a cutoff → Supported with its boundary, if warranted.

Claim: “The program increased employment near the eligibility cutoff.” Suppose a regression discontinuity analysis supplies defensible continuity and allocation reasoning, appropriate measurement and uncertainty, and checks for manipulation, other cutoff policies and sensitivity. Finding: if the estimate supports the direction, state it for people near that cutoff and the studied period. Passing diagnostic tests does not prove every identifying assumption. If eligibility only changes the probability of receipt, an offer effect and an effect of participation require different interpretations.

5.4

4 · An outcome claim with no outcome record → Unsupported as stated.

Claim: “We transformed the lives of 10,000 people.” Checked records show 10,000 unique recipients in the reporting period; outcomes were not measured. Finding: “transformed lives” is unsupported. A separate delivery sentence, “We delivered the service to 10,000 people,” can be supported within the definitions and period of those records.

6What this review records

The review assesses the support for a sentence and the limits that must travel with it. A supported finding can be strong and favorable; uncertainty should be stated accurately rather than used to suppress it.

Complete: original sentence / evidence job / construct and unit / people, version and period / source and coverage / warrant and assumptions / uncertainty and contrary evidence / review outcome / revised sentence and essential limit / reviewer, date and disagreements / unresolved task, cost, permission and owner / intended use and actual decision authority / review trigger.

For example 1, the unresolved task is to assess selection and a feasible causal strategy, if attribution matters to the intended use. That work is not commissioned by a wording review. A renewal authority may need a bounded present account before stronger inquiry is feasible.

The same checks apply to Prova's own claims. Reopen the record when sources, definitions, conditions or relevant evidence change.

Related: The Identification Atlas, for identifying comparisons; The Risk Taxonomy of Evaluation Failure, for the ways an evaluation can fail; The Claim Ladder, for the map of evidence jobs.

Further reading

  1. 1
    World Bank/IDB, Impact Evaluation in Practice (2016), chapters 3–8 and missing follow-up: counterfactual, assignment and method-specific assumptions.
  2. 2
    HM Treasury, Magenta Book Annex A, A1.3–4: causal mechanism and contribution inquiry.
  3. 3
    Spencer and colleagues, Quality in Qualitative Evaluation (2003; Treasury republication 2012), questions 1–2 and 12–14: evidence, contrary accounts and traceable interpretation.