Decide who settles unforeseen questions before an unwelcome result arrives
No protocol anticipates every choice. Data arrive late. A source changes. Eligibility is applied differently than expected. A subgroup raises a safety concern. An analysis produces conflicting estimates.
If the agreement does not say who holds authority, those questions are often settled informally by the person paying, the person with the strongest relationship, or the person under the greatest pressure from the result.
The short answer
Allocate methodological authority before the work begins. Distinguish factual correction, operational input, analytic judgment, legal or ethical constraint, and final decision authority. Keep a dated log of material choices and preserve dissent when responsible parties disagree.
Independence depends partly on who can say no when a requested change would make the conclusion less trustworthy.
Different kinds of authority
Factual authority
Providers and participants may be best placed to correct names, dates, delivery descriptions, and contextual errors. Factual correction should not become control over interpretation.
Operational authority
Delivery leaders decide what can happen safely and practically in the program. Their knowledge should shape design, but operating convenience alone should not determine the analysis.
Methodological authority
The scope should allocate responsibility for translating the agreed question into definitions, comparisons, exclusions, models, uncertainty and claims. A protected assessor role is the arrangement proposed here, not a power automatically granted by an external evaluator label. This authority carries a duty to explain choices and consider valid challenge.
Legal and ethical authority
Privacy, consent, safeguarding, rights, and statutory duties can constrain methods. These are not optional preferences to be traded for a cleaner design.
Decision authority
The board, funder, executive, public official, or shared body decides what to do with the findings. This role does not automatically confer authority to rewrite how the finding is produced.
Make the boundaries between these roles explicit.
Questions that often arrive late
- Should a newly available outcome replace the planned one?
- Can cases with incomplete records be excluded?
- How should crossover or partial delivery be handled?
- Does a subgroup analysis belong in the main report?
- Can the commissioner request a more favorable comparison period?
- Should a safety concern be reported outside the planned process?
- What happens when provider and assessor disagree about a factual description?
- Who decides whether uncertainty is too great for the proposed claim?
For each class of question, name who decides, who must be consulted, and where disagreement is recorded.
A worked example
An agency commissions an evaluation of a contracted employment service. Midway through analysis, the agency asks the evaluator to exclude participants referred during a period of unusually high unemployment. The provider supports the request because outcomes were poor. The original scope says only that the parties will agree analytic changes together.
The people, places, organizations and figures in this document's examples are fictional.
The evaluator examines the issue and concludes that excluding the period would remove a real part of program delivery and make the headline result less representative. Under the proposed charter, the agency can request the analysis, but the evaluator holds methodological authority. The final report retains the full-period estimate, adds a clearly labeled sensitivity analysis, and records the disagreement.
The agency still owns the funding decision. It does not own the answer.
Create an authority charter
A short charter can sit inside the scope or protocol. It should state:
- Under the proposed charter, the assessor has protected responsibility for methodological conclusions within the agreed question. Verify the arrangement and adopt it locally.
- The provider can correct factual inaccuracies and explain delivery context.
- The commissioner can clarify the decision and request additional analyses.
- Material changes are logged with timing, reason, and result visibility.
- Legal, ethical, and safeguarding constraints override convenience.
- Dissent is recorded when agreement is not reached.
- No party can suppress a specified finding solely because it is unfavorable.
- The decision-maker remains responsible for action after receiving the conclusion.
The wording can vary. The structure should not depend on goodwill alone.
Keep a decision log
For every material issue, record:
- 1The question.
- 2When it arose.
- 3What information and results were visible.
- 4Options considered.
- 5Who was consulted.
- 6Who decided and under what authority.
- 7The reason.
- 8Any dissent.
- 9Effect on interpretation or reporting.
The log protects the work from silent drift and helps a later reviewer reconstruct what happened.
What you can say
- "The assessor holds methodological authority. The provider can correct factual descriptions."
- "The commissioner requested this additional analysis, which is labeled exploratory."
- "The parties disagreed, and both the decision and dissent are recorded."
- "The board owns the response to the evidence, not the analytic conclusion."
What you cannot say yet
- "The review is independent," when the commissioner can require changes to the conclusion.
- "All parties agreed," when disagreement was resolved through funding power.
- "The change was technical," when it materially altered the result or claim.
- "The methodology was fixed," when important residual choices had no owner.
What to do next
Minimum useful step: Identify who currently decides unforeseen methodological questions.
Stronger step: Add an authority charter and decision log to the engagement. Separate factual correction, consultation, requested analysis, and control over conclusions.
When independence matters: Ensure the reviewer can decline requests, report dissent, and release the agreed conclusion without a favorable-outcome condition.
A 15-minute team exercise
List five issues that could arise after work begins: missing data, changed outcome, delivery deviation, new subgroup, and disputed wording.
- 1Ask each person who they believe decides each issue.
- 2Compare answers and mark conflicts.
- 3Classify each issue as factual, operational, methodological, legal or ethical, or decision authority.
- 4Name who must be consulted and who has final authority.
- 5Agree how dissent will be recorded.
- 6Add the result to the scope or protocol.
- 7Assign ownership for the decision log.
If the answer to every issue is "we will agree," the hard question remains unresolved: what happens when you do not?
A note on relationships
Evidence work is relational. Informal resolution can be fast and humane. It can also make it difficult for an assessor, provider, or participant representative to resist the party controlling money or access.
A clear authority arrangement does not remove collaboration. It makes collaboration safer by showing which decisions are shared and which responsibilities should remain separate.
When to bring in help
Independent review is useful when authority has been disputed, material changes occurred after results were visible, or the same party controls commissioning, analysis, and release. Legal or ethical expertise may be needed where the methodological choice affects rights, privacy, or safety.
The useful output is not a declaration that one party was right. It is a traceable account of who decided what, under which authority, and how that choice affects the conclusion.
Questions to carry forward
- Who can make a methodological decision when the plan is silent?
- Which changes require consultation, approval, or independent judgment?
- How will dissent remain visible?
- Can the assessor refuse a change that would weaken the integrity of the conclusion?
A completed record, with room for your own case
Read the completed example first. Then fill in the last column with your own facts and permissions. Where information is missing, mark it unresolved.
| Field | Completed example | Your case |
|---|---|---|
| Situation and evidence | After the outcomes became visible, someone asked to exclude a period of high unemployment. | |
| Responsibility and next task | Under the charter, the assessor keeps the finding for the full period, labels the shorter period as a sensitivity check and records the dissent. The agency decides the funding. | |
| Timing and dependencies | Before the analysis changes, log the request, when the results became visible, who has authority and the reasons. | |
| Limit on the claim or action | A confirmed data error can justify a correction. Protected judgment means weighing a valid challenge on its merits, not rejecting every request from the commissioner. | |
| Review and dissent | Record the evidence against the conclusion, the interpretations still unresolved, the response that has been authorized and what will trigger a review. |
Try: suppose the condition in the row "Limit on the claim or action" changed. Which conclusion would change, and which would stay the same?
Show the feedback
A confirmed data error can justify a correction. Protected judgment means weighing a valid challenge on its merits, not rejecting every request from the commissioner. Keep the original finding, the scope of the revised claim and the authority for the next action apart. Use the 15-minute team exercise to apply this to your own case. The completed record shows the reasoning, but it does not approve anything for your organization.
Related: Ten Questions to Ask Before You Sign the Contract, for what to ask before agreeing terms; How Much Evidence Is Enough?, for deciding how much evidence a decision needs.
Further reading
- 1American Evaluation Association, Guiding Principles. Principles A2–A5, C1–C7, D1–D4 and E1–E4: inquiry, integrity, people and equity.
- 2GAO, Assessing Data Reliability, GAO-20-283G (2019). The official overview: data accuracy, completeness and applicability to purpose.
- 3Spencer, Ritchie, Lewis and Dillon, Quality in Qualitative Evaluation (2003; Treasury republication 2012). Appraisal questions 1–2 and 12–14: evidence, contrary accounts and traceable interpretation.