← MethodsSCREEN READING EDITIONDownload the PDF

Who Holds Methodological Authority?

i

Decide who settles unforeseen questions before an unwelcome result arrives

For:commissioners, funders, providers, evaluators, analysts, boards, and partners entering evidence agreements
Use when:a scope specifies tasks but not who decides exclusions, outcome changes, missing-data treatment, interpretation, or other unanticipated issues
Includes:an authority map, decision log, dissent process, worked example, and a 15-minute team exercise
Helps when you are:reviewing an evaluation design, analysis, report or claim, starting new work, or working out whose decision or learning the work serves

No protocol anticipates every choice. Data arrive late. A source changes. Eligibility is applied differently than expected. A subgroup raises a safety concern. An analysis produces conflicting estimates.

If the agreement does not say who holds authority, those questions are often settled informally by the person paying, the person with the strongest relationship, or the person under the greatest pressure from the result.

ii

The short answer

Allocate methodological authority before the work begins. Distinguish factual correction, operational input, analytic judgment, legal or ethical constraint, and final decision authority. Keep a dated log of material choices and preserve dissent when responsible parties disagree.

Independence depends partly on who can say no when a requested change would make the conclusion less trustworthy.

iii

Different kinds of authority

Factual authority

Providers and participants may be best placed to correct names, dates, delivery descriptions, and contextual errors. Factual correction should not become control over interpretation.

Operational authority

Delivery leaders decide what can happen safely and practically in the program. Their knowledge should shape design, but operating convenience alone should not determine the analysis.

Methodological authority

The scope should allocate responsibility for translating the agreed question into definitions, comparisons, exclusions, models, uncertainty and claims. A protected assessor role is the arrangement proposed here, not a power automatically granted by an external evaluator label. This authority carries a duty to explain choices and consider valid challenge.

Legal and ethical authority

Privacy, consent, safeguarding, rights, and statutory duties can constrain methods. These are not optional preferences to be traded for a cleaner design.

Decision authority

The board, funder, executive, public official, or shared body decides what to do with the findings. This role does not automatically confer authority to rewrite how the finding is produced.

Make the boundaries between these roles explicit.

iv

Questions that often arrive late

For each class of question, name who decides, who must be consulted, and where disagreement is recorded.

v

A worked example

An agency commissions an evaluation of a contracted employment service. Midway through analysis, the agency asks the evaluator to exclude participants referred during a period of unusually high unemployment. The provider supports the request because outcomes were poor. The original scope says only that the parties will agree analytic changes together.

The people, places, organizations and figures in this document's examples are fictional.

The evaluator examines the issue and concludes that excluding the period would remove a real part of program delivery and make the headline result less representative. Under the proposed charter, the agency can request the analysis, but the evaluator holds methodological authority. The final report retains the full-period estimate, adds a clearly labeled sensitivity analysis, and records the disagreement.

The agency still owns the funding decision. It does not own the answer.

vi

Create an authority charter

A short charter can sit inside the scope or protocol. It should state:

The wording can vary. The structure should not depend on goodwill alone.

vii

Keep a decision log

For every material issue, record:

  1. 1
    The question.
  2. 2
    When it arose.
  3. 3
    What information and results were visible.
  4. 4
    Options considered.
  5. 5
    Who was consulted.
  6. 6
    Who decided and under what authority.
  7. 7
    The reason.
  8. 8
    Any dissent.
  9. 9
    Effect on interpretation or reporting.

The log protects the work from silent drift and helps a later reviewer reconstruct what happened.

viii

What you can say

ix

What you cannot say yet

x

What to do next

Minimum useful step: Identify who currently decides unforeseen methodological questions.

Stronger step: Add an authority charter and decision log to the engagement. Separate factual correction, consultation, requested analysis, and control over conclusions.

When independence matters: Ensure the reviewer can decline requests, report dissent, and release the agreed conclusion without a favorable-outcome condition.

xi

A 15-minute team exercise

List five issues that could arise after work begins: missing data, changed outcome, delivery deviation, new subgroup, and disputed wording.

  1. 1
    Ask each person who they believe decides each issue.
  2. 2
    Compare answers and mark conflicts.
  3. 3
    Classify each issue as factual, operational, methodological, legal or ethical, or decision authority.
  4. 4
    Name who must be consulted and who has final authority.
  5. 5
    Agree how dissent will be recorded.
  6. 6
    Add the result to the scope or protocol.
  7. 7
    Assign ownership for the decision log.

If the answer to every issue is "we will agree," the hard question remains unresolved: what happens when you do not?

xii

A note on relationships

Evidence work is relational. Informal resolution can be fast and humane. It can also make it difficult for an assessor, provider, or participant representative to resist the party controlling money or access.

A clear authority arrangement does not remove collaboration. It makes collaboration safer by showing which decisions are shared and which responsibilities should remain separate.

1

When to bring in help

Independent review is useful when authority has been disputed, material changes occurred after results were visible, or the same party controls commissioning, analysis, and release. Legal or ethical expertise may be needed where the methodological choice affects rights, privacy, or safety.

The useful output is not a declaration that one party was right. It is a traceable account of who decided what, under which authority, and how that choice affects the conclusion.

14

Questions to carry forward

15

A completed record, with room for your own case

Read the completed example first. Then fill in the last column with your own facts and permissions. Where information is missing, mark it unresolved.

FieldCompleted exampleYour case
Situation and evidenceAfter the outcomes became visible, someone asked to exclude a period of high unemployment.
Responsibility and next taskUnder the charter, the assessor keeps the finding for the full period, labels the shorter period as a sensitivity check and records the dissent. The agency decides the funding.
Timing and dependenciesBefore the analysis changes, log the request, when the results became visible, who has authority and the reasons.
Limit on the claim or actionA confirmed data error can justify a correction. Protected judgment means weighing a valid challenge on its merits, not rejecting every request from the commissioner.
Review and dissentRecord the evidence against the conclusion, the interpretations still unresolved, the response that has been authorized and what will trigger a review.

Try: suppose the condition in the row "Limit on the claim or action" changed. Which conclusion would change, and which would stay the same?

Show the feedback

A confirmed data error can justify a correction. Protected judgment means weighing a valid challenge on its merits, not rejecting every request from the commissioner. Keep the original finding, the scope of the revised claim and the authority for the next action apart. Use the 15-minute team exercise to apply this to your own case. The completed record shows the reasoning, but it does not approve anything for your organization.

Related: Ten Questions to Ask Before You Sign the Contract, for what to ask before agreeing terms; How Much Evidence Is Enough?, for deciding how much evidence a decision needs.

Your entries are saved in this browser only.
16

Further reading

  1. 1
    American Evaluation Association, Guiding Principles. Principles A2–A5, C1–C7, D1–D4 and E1–E4: inquiry, integrity, people and equity.
  2. 2
    GAO, Assessing Data Reliability, GAO-20-283G (2019). The official overview: data accuracy, completeness and applicability to purpose.
  3. 3
    Spencer, Ritchie, Lewis and Dillon, Quality in Qualitative Evaluation (2003; Treasury republication 2012). Appraisal questions 1–2 and 12–14: evidence, contrary accounts and traceable interpretation.