A review creates an answer through decisions about which question to ask, which evidence to include and how to combine it. Those decisions deserve scrutiny of their own.
A foundation receives two reviews of youth-employment programs. One concludes that the evidence is encouraging. The other finds little support for a sustained earnings benefit. Both cite substantial research. Both describe their process as systematic.
They may disagree because they made different choices before reading the first result.
In this example, the first review examines wage subsidies and employment at six months. The second includes training, placement and mentoring, and asks about earnings two years later. Their conclusions concern different interventions, outcomes and time horizons. Counting their citations will not resolve the difference.
The people, places, organizations and figures in this document's examples are fictional.
A systematic review is a study because it has a research question, a design, an observable evidence base and an analysis. Its conclusion is produced by that process.
Decide what evidence would answer the question
Before commissioning new data, it is worth establishing what existing studies can contribute. The U.S. GAO's Designing Evaluations, printed pp. 21–22, recommends considering synthesis when previous studies address the evaluation question. It also treats differences across studies as something to investigate: variation in implementation or populations may help explain variation in findings.
That is a useful starting point for the foundation. It needs to specify which decision the synthesis will inform. Renewing a wage-subsidy grant, choosing among employment services and estimating long-term economic gains are different tasks.
The protocol should settle the important boundaries before findings make one boundary more attractive than another:
- the population and settings of interest;
- the intervention and relevant alternative;
- outcomes and time horizons;
- which designs can answer each question; and
- how applicability, bias and conflicting findings will be judged.
A protocol does not prevent learning during a review. It makes changes visible. If the team broadens eligibility after discovering few relevant studies, the reader should know what changed, why it changed and how that affects the answer.
The search defines what the review can see
A search limited to published journal articles may miss government evaluations, trial records, dissertations or technical reports. Restricting language or geography can change which populations and delivery settings are represented. Different disciplines may describe similar services in different terms.
These choices affect the body of evidence available for the conclusion. “We found twelve studies” is incomplete without an account of what the search could have missed.
There is also a counting problem. One evaluation may produce a working paper, a journal article, a registry entry and several follow-up reports. The Cochrane Handbook's chapter on searching and selecting studies distinguishes a study from its reports and calls for linking reports of the same study. Four documents from one trial are not four independent replications. Equally, a later report may contain outcomes missing from the first publication.
For the foundation, the practical request is a study-to-report map: which documents belong to which underlying evaluation, which findings were extracted from each, and which reports remain unavailable. This helps a reviewer detect both double counting and missing information.
Combine findings only after checking what they represent
A meta-analysis can make an estimate more precise without making the underlying studies more credible. Statistical precision does not repair selective reporting, biased comparisons or incompatible outcomes.
Before pooling, the review needs to ask whether the findings represent sufficiently similar quantities. An effect of being offered training differs from an effect among people who completed it. Employment at four weeks differs from sustained earnings at two years. A large study can contribute considerable statistical weight while still having a serious risk of bias.
Some differences are reasons to separate the estimates. Others are hypotheses to investigate. A structured narrative may be more useful than a single pooled number when interventions or outcomes are too different to combine defensibly.
The Magenta Book's synthesis guidance, sections A4.1–A4.2, describes systematic reviews and rapid evidence assessments as methods with different demands and limitations. The label does not determine whether the method fits the decision. A short, clearly bounded review may be useful; an expansive review can still answer the wrong question.
Judge confidence for each consequential outcome
The foundation could have credible evidence of increased enrollment, uncertain evidence of completion and little evidence about earnings. Calling the whole program “well supported” would conceal where the support stops.
The GRADE framework separates estimated effects from certainty in the evidence and considers risk of bias, inconsistency, indirectness, imprecision and publication bias. It originated in health research; selecting or adapting an appraisal approach for a policy review requires care. The practical lesson is to assess confidence for each decision-critical conclusion.
The evidence can also be credible and still be insufficient for a recommendation. Costs, feasibility, distribution of benefits and harms, and the available alternatives belong in the decision. A synthesis should make clear which of those questions it answers and which remain open.
Ask for a review receipt
A decision-maker should be able to obtain a concise account of how the review reached its conclusion. It need not repeat the technical appendix. It should expose the choices most capable of changing the answer.
- 1Question and scope: population, intervention, comparator, outcomes, time horizon and intended decision.
- 2Search and coverage: sources searched, search date, restrictions and likely missing evidence.
- 3Selection and extraction: eligibility decisions, handling of multiple reports, extracted results and how checking or disagreements were managed.
- 4Appraisal and synthesis: risk-of-bias judgments, reasons for pooling or separating studies, and treatment of conflicting findings.
- 5Conclusion and limits: confidence by outcome, applicability to the intended setting, and what remains unknown.
- 6Accountability: protocol and deviations, funding and conflicts, review team, and access to the underlying records where appropriate.
This is Prova's suggested commissioning receipt. Its purpose is to help someone follow the argument from question to evidence to conclusion.
Reporting frameworks can make that inspection easier. PRISMA 2020 provides a reporting checklist and flow diagrams. It is not a certificate of sound methods: a review can transparently report weak procedures. Readers still need to judge the choices disclosed.
Make rapid work explicit
A decision due in three weeks may not allow a full systematic review. The useful response is to agree which questions can be answered credibly in the time available and document the concessions.
A team might narrow the population, prioritize a set of outcomes or limit the search. Each concession should have a reason and an account of what it may omit. Both a systematic review and a rapid review can expose their methods; the difference is not that one is automatically transparent and the other cannot be.
For the foundation, a transparent rapid review might support a provisional renewal while identifying a longer-term earnings question for further work. It should not acquire the appearance of certainty simply because its report looks complete.
Before relying on the conclusion, select one important finding and trace it backward: to the extracted result, the underlying study, the eligibility rule and the question the review was designed to answer. A useful synthesis makes that journey possible.
Related: The Trial Tested the Offer, for why similar effect estimates can answer different questions; Human Review Has to Be Tested, for making the checking of an evidence product a working control.
Further reading
- 1U.S. GAO, Designing Evaluations, 2012 revision, printed pp. 21–22 — using existing studies before collecting new data and examining differences across findings.
- 2HM Treasury, Magenta Book, Annex A (2026), sections A4.1–A4.2, pp. 43–45 — systematic reviews and rapid evidence assessments.
- 3Cochrane, Handbook, chapter 4: Searching for and selecting studies, especially section 4.6.2 — connecting multiple reports of one study.
- 4PRISMA 2020 statement — transparent reporting of a systematic review.
- 5GRADE Handbook — judging certainty in a body of evidence and separating evidence judgments from recommendations.