What this standard judges
Someone wants to put "independently evaluated" in a board paper, a report or a press release. This standard helps you decide whether that wording is earned. It covers one job: judging an independence claim about a specific evaluation or review after the work has begun, using what actually happened as well as what the terms said.
This standard uses the definition in Structuring an Evidence Function. Independence concerns an evaluator's position and freedom to do the work and report it. Impartiality concerns how people approach the inquiry and treat the evidence. Competence concerns whether the people and the design were up to the job. The three are related, and a judgment about one cannot stand in for the others. A protected team can still use a weak design, and a skilled team can still work under terms that let someone suppress a finding. This standard judges independence and asks you to report impartiality and competence beside it.
Being an outside firm does not settle the question, and neither does a confidentiality label or a fixed fee. A staff team can be independent for a given piece of work, and an outside firm can be under pressure. To choose an arrangement before work starts, use Who Should Judge Our Program? Choosing an Independent Arrangement.
Set the boundary of your review
Write down the exact claim, such as "the evaluation of the reading program was independent" or "a separate reviewer checked the analysis." A protected review of one analysis is a much narrower claim than an independently conducted evaluation. Name the work, the period and the use the claim will serve.
Say who is reviewing, how they relate to the work, what interests they have, what they are qualified to judge and what access they have. If the original author checks their own work a second time, that check can find problems, and it still is not an independent review.
Inspect the records that show what happened. Start with the terms of the commission and any conflicts people declared. Then look at who held the data, the plan and its changes, and the review comments. Last, check payments, each version of the report, when it went out and how concerns were answered. Use protected access where records identify people. Checking the work does not mean sharing records that name people with everyone.
Keep the documented arrangement apart from how it worked. Terms show what people agreed to. Only the record of decisions, comments, changes, payments and circulation shows whether the protection held under pressure. Where policy and conduct disagree, record both.
The seven criteria
For each criterion, the standard gives what it requires, what to inspect and what to do if the requirement is not met.
1The claim and the roles are clear
What it requires.
The work states its question, its purpose and how much independence it claims. Help with improvement, assessment of evidence, independent judgment and the institution's own decision are kept apart.
What to inspect.
Look at the brief, the scope, any public wording, the role descriptions and the actual products. Check whether an outside review of wording is being presented as an independent evaluation of how data were collected and analyzed.
If it is not met.
Define or narrow the claim before anyone uses it. Internal learning can continue under its own name. Relabeling it will not supply the separation that an important judgment needs.
2Interests were disclosed and dealt with
What it requires.
The interests of the sponsor, the assessor, any reviewer and anyone who produced important data are known. That includes earlier roles in designing or running the program and hopes of future work. The response fits the judgment required.
What to inspect.
Look at the disclosures, the reasons for the appointment, any separation of roles and later changes. Ask what each safeguard actually protected, and whether anyone ended up judging their own design.
If it is not met.
Get the missing information, change roles, add capable scrutiny or narrow the claim. Disclosure alone does not resolve a conflict. An earlier role does not rule someone out automatically, and an outside appointment does not clear them automatically.
3There was a working route for raising interference
What it requires.
The assessor could raise interference with someone able to act, and that route did not depend on the program's champion agreeing.
What to inspect.
Look at who the assessor reported to, who could receive concerns and whether any concern was raised and answered. Test what happened when people disagreed.
If it is not met.
Record that protection was not available, and narrow the claim. A committee with no remit or power to act does not count as a route. A small organization can use a simpler route, as long as the record states its limits.
4Nobody with a stake could quietly change the data
What it requires.
You can trace the result back to the events it describes. People know who defined, recorded, corrected, extracted and analyzed the data, and reviewers had access suited to the claim.
What to inspect.
Look at the source history, definitions, coverage, corrections, exclusions and access limits. Find out who could change the outcome at each important point.
If it is not met.
Resolve the access or source question, add a proportionate check, or narrow the claim. Holding the data independently does not make the measure valid, and a reviewer cannot recover outcomes nobody observed.
5Authority over method and changes was clear
What it requires.
One named party settled unexpected questions about method, considered challenges and explained conclusions. Scope, money and safety decisions had their own owners. Important changes can be traced.
What to inspect.
Look at the plan, the record of changes, the reasons given, any dissent and which results people could see when each choice changed. For a study planned in advance to test a specific question, find the original commitments.
If it is not met.
Record who actually decided and what that means for the conclusion. Do not backdate a plan, and do not let a favorable subgroup quietly replace the planned answer. A verified error can still be corrected.
6Payment depended on the work, not the result
What it requires.
Acceptance and payment depended on agreed work and quality. A process existed for real disputes about quality or scope.
What to inspect.
Look at the payment terms, the acceptance criteria and any payment that was withheld or changed, with the reasons. Check for hopes of future work. An unfavorable or inconclusive finding can be a properly completed job.
If it is not met.
Refer the terms or conduct to the contract and oversight owners. This standard does not itself authorize payment, withholding or an amendment.
7The findings reached the people they were meant for
What it requires.
The terms named recipients, products, timing, grounds for review, corrections, dissent and justified restrictions. Nobody with a stake removed a supported finding because they disliked it.
What to inspect.
Compare the full and short versions, check circulation dates and read the release decisions. See whether a qualification disappeared from the summary, or a delay meant the finding missed the decision it was for.
If it is not met.
Repair the reporting route through its owners and keep the version history. Keep private details safe by limiting who sees them, by grouping figures, or by letting people read them only under control. A dispute alone is no reason to make a report public. Being open never means publishing private records about participants.
Report impartiality and competence beside the judgment
Two questions sit next to independence, and your record should answer them separately.
Was the evidence treated fairly? Check that good, bad and unclear findings got the same scrutiny. See whether fair challenges were weighed on their merits, and whether dissent was kept. Refusing a correct correction from a provider in order to look independent is as unfair as accepting an unsupported favorable rewrite.
Was the work competent? Check that the people who did the work had the right skills, and that review covered the question, design, data, analysis and conclusion as far as the claim needs. A senior name on a proposal does not show the working team's skill, and a review of the prose alone cannot vouch for the method. A well-protected but unsuitable design still cannot support a causal conclusion.
Record each criterion without a total score
For each criterion, choose one of four findings and give the reason and where the evidence is.
- Supported for this scope. The records establish the protection. Say whether you saw it in the terms only, or in what happened too.
- Unresolved. Evidence or authority needed for the judgment is missing or unclear. Name the gap and the next step.
- Contradicted. The terms or conduct conflict with the protection. Describe the conflict, without guessing at motives.
- Outside this review. You did not assess it. Explain why, and narrow the overall claim. This finding cannot be used to skip a condition the claim depends on.
Do not add up favorable rows or set a pass mark. One power to suppress the conclusion can defeat the claim even when everything else is sound. A small gap in the paperwork may still allow a narrower, accurately worded claim.
Match the overall judgment to the record
There are three possible overall judgments.
The claim is supported within this review.
Say which protections were established, what you inspected, whether you saw them operate and what limits remain. Describe the scope, and avoid an unrestricted label such as "fully independent."
A narrower claim is justified.
Say what is supported and which wider claim has to wait. "A separate reviewer checked the analysis" may be accurate when "the evaluation was independent" is not.
The claim is not established.
Name the blocking gaps or contradictions, the work affected and who owns the repair. This judgment does not show the finding is wrong. It shows the independence wording cannot be used for the intended purpose on this record.
Record your reasons and any disagreement. The standard supplies criteria. It gives nobody the power to appoint, contract, publish or decide the institution's response.
Three worked judgments
The people, places, organizations and figures in this document's examples are fictional.
An outside firm whose conclusions needed the program director's approval.
Halfway through, the reviewer finds that the terms give the program director final say over the analysis wording and final payment. The director could stop an unfavorable finding from reaching the board. Criteria 5, 6 and 7 are contradicted. Being external and having good data access do not make up for that. The claim is not established. The owners need to repair the terms, or the foundation needs to narrow its wording and reconsider how it uses the report.
An internal unit with a gap in its data.
The foundation's evaluation unit reports separately from program management, is funded for the work, has a working route to a board committee and can send agreed findings to the board. The records show those routes operated during this evaluation. The main outcome source misses some families who moved away. The claim is supported for this scope, with the measurement gap reported beside it. Independence does not turn an incomplete source into a complete one.
A commission that has only just begun.
The terms set out clear roles, access, payment and reporting, and analysis started last month. The reviewer can confirm what the terms say and cannot yet say the protection held under pressure. The record supports the narrower claim that the commission's terms provide these protections. A named owner will recheck at the first major change and when the report circulates.
If your organization is the one being evaluated
Executive directors can use the same criteria when a funder plans to call an evaluation of their program independent. Ask to see the terms on criteria 4 to 7: who controls your data in the analysis, who decides questions of method, what payment depends on and who receives the findings. Correcting facts and challenging methods through the agreed route is part of a fair process, and your corrections should be weighed on their merits.
If you commissioned the evaluation yourself, be careful with the word "independent" in your own reports. If program staff could require changes to supported methodological conclusions or prevent agreed reporting, do not make an unrestricted independence claim. Describe the actual reporting relationship, review powers and limits. Distinguish factual correction, accessibility and legitimate confidentiality checks or administrative approval from a substantive veto over findings. Merely calling the work external does not remedy interference; a precise account should say what happened.
Keep the basis and make repairs checkable
Keep the following.
| What to record | Your entry |
|---|---|
| The work reviewed and the period | |
| The exact independence claim | |
| The reviewer, their qualifications and interests | |
| Records inspected, and access limits | |
| The finding for each criterion, with where the evidence is | |
| What the terms said, and what happened | |
| Notes on impartiality and competence | |
| The overall judgment | |
| Disagreement | |
| Repairs, their owners and who authorized them | |
| The claim that can be made now | |
| The recheck date or trigger |
For each repair, say what evidence will show it is done, and get approval before describing revised terms as in force. Recheck when roles, funders, data sources, payment, scope or release arrangements change, and whenever conduct calls the documented protection into question.
Related: Who Should Judge Our Program? Choosing an Independent Arrangement, for choosing an independent arrangement before the work starts; Structuring an Evidence Function, for the definitions of independence and impartiality and for governing evidence work; Who Holds Methodological Authority?, for who has the final say on method; Who Controls the Outcome Measurement?, for checking who can change what is counted and how; What Happens to Unfavorable Findings?, for agreeing in advance how a poor result will be handled.
Further reading
- 1UNEG, Norms and Standards for Evaluation, 2016; updated reprint 2017, norms 4–5, printed page 11: behavioral and organizational independence, and impartiality.
- 2American Evaluation Association, Guiding Principles for Evaluators, principles C (Integrity) and D (Respect for People): for disclosing interests and limits, documenting changes and treating the people involved fairly.