Knowing what works should not be a luxury.
Today it is one. A program can learn how much it helps, and for whom, in several ways: by comparing results over time, by running a careful study, by listening to the people it serves, or by keeping good records. Each of these has limits.
The federal government gave state and local governments an estimated $1.1 trillion in grants in fiscal year 2024.1
In 2013, two former White House officials estimated that less than $1 of every $100 of government spending is backed by even the most basic evidence that it is spent well.2
In a 2016 survey, 84 percent of the 740 nonprofits that answered spent less than 5 percent of their budget on evaluation.3
People should be able to rely on what is said about public work.
Other fields built institutions that let people rely on a claim made by someone they would never meet.
Finance built the independent audit. Before it, money went mostly where people could see for themselves: to family, to neighbors, and to firms they knew. After the securities laws of the 1930s required audited accounts, an investor in another state could rely on a company’s numbers without ever visiting it.
The Securities Act of 1933 required audited financial statements from companies selling securities to the public.4
Medicine built the controlled trial, so a doctor could prescribe a treatment that had been tested on patients she would never meet.
In 1962, Congress required drug makers to show substantial evidence of effectiveness from adequate and well-controlled studies.5 A landmark controlled trial of streptomycin had been published in 1948.6
Industry built the testing laboratory, so a buyer could trust a part made by someone far away.
Congress created the National Bureau of Standards in 1901, and it tested materials for industry and government.7
Each of these institutions has a narrow job. It tells people which claims they can rely on, and that allows people to trust a stranger’s claim.
Public and social programs have no institution like this at a price an ordinary program can pay.
Few people can check how much a program helps, but anyone can check the number of people enrolled, the hours delivered, or the meals served. So grants and contracts are written around those counts, and reports fill with them.
In a 2013 survey answered by 2,762 federal managers, 37 percent said a program they worked on had been evaluated in the past five years, and another 40 percent did not know.8
A bipartisan federal commission reported in 2017 that “too little evidence is produced to meet this need.”9
Economists have described the gap this leaves. In 2016 the Nobel prize went to work on contracts, which showed how agreements weaken when what matters cannot be observed or checked, and how incentives drift toward whatever can be measured.10, 11 In 2019 it went to an experimental way of learning what reduces poverty.12 Together, the two pieces of work describe why programs need a trusted way to check claims.
Much of what programs learn also leaves with the people who learned it.
In a 2024 survey of more than 350 nonprofit CEOs, one in three said they were likely to leave their organization within two years.13
In the 2022 Race to Lead survey, reported in 2024, 22 percent of executive directors of color and 30 percent of white executive directors received support from their predecessor.14
If claims about programs could be checked, a small program with no famous name or connections could show what it does for people, and the people it serves could check what is said about them.
Price was one of the main obstacles, and parts of the work now cost less.
A credible study took years, cost more than most programs could raise, was usually paid for by the programs being studied, and produced a result that could rarely be compared with any other. Nothing like that could be written into an ordinary grant.
Across 70 evaluation plans in one federal grant program, the 15 studies aiming at strong evidence averaged $589,037 a year, more than a quarter of the program’s budget. The usual rule of thumb allows five to ten percent.15
In the same 2016 survey, evaluation at only 8 percent of nonprofits was led by evaluation staff, whether employed or hired.3
Most of the methods a credible study needs have been settled for decades, and they can be written down once and used by many programs. MDRC has studied social programs since 1974, and researchers affiliated with J-PAL have led more than 2,700 randomized evaluations in more than 140 countries.16, 17 Much of what a program needs to learn from is already in its own files.
A smaller question can often come first, and it costs far less.
Before anyone asks how much a program helps, it is worth asking whether that question can be answered at all from what already exists, and with how much confidence.
Sometimes a careful study is already within reach. When a program has 300 places and 900 people apply, a fair lottery gives every applicant the same chance, as long as applicants are told how places are chosen, and it also creates a fair comparison between those who received a place and those who did not.
Language models can now take on part of the reading, source tracing, and checking, with a person checking the result. People still make the judgments, do the fieldwork, and speak for their own experience.
In a 2025 study within six systematic reviews, data extraction assisted by a language model and checked by people was correct 91 percent of the time, against 89 percent for people working alone, and it saved a median of 41 minutes per study.18
Governments have started to ask for it. The Foundations for Evidence-Based Policymaking Act of 2018 asks federal agencies to plan for and use evidence.19 Each agency must send Congress and the Office of Management and Budget a plan for the questions it needs evidence to answer, and name a senior evaluation officer.19
Prova is starting with a library and a small practice.
Prova works to make measurement, evaluation, and learning affordable for nonprofits and state and local governments.
The library holds collections of documents, each built around one decision, such as a renewal case or a review of a disappointing result. Some collections are written for communities and the people programs serve, to help them ask their own questions of the institutions that serve them. Its method notes and standards are free to read in Methods. Ravi Narain also takes on a small number of engagements with foundations and public agencies.
No single organization can build an institution like this, so Prova’s methods are free for any organization to read and use.
Prova’s work has seven purposes.
Evidence an ordinary program can afford
Important questions get serious help, even when no one can pay for it.
- Why
- Today, whether a question gets investigated depends mostly on budget and access to a research team. A nonprofit may know how many families it served and still not know why other families never reached it.
- How
- Methods are written down once and reused, language models do much of the reading and checking, and a team gets help turning a problem it sees into a question that can be answered. Sometimes the most useful help is to say what the evidence cannot yet show.
- How we will know
- People who could not investigate a worthwhile question before are able to, and they can understand and question what was found.
Trust that rests on evidence
People can check claims about public work for themselves.
- Why
- Trust in a claim should rest on a report people can examine.
- How
- Each claim is stated only as strongly as its evidence allows, and uncertainty stays visible. Findings that disappoint the people who paid for the work stay in the report, and corrections reach the people who relied on the earlier claim.
- How we will know
- People can understand, question, and use a report that matters to them.
Learning that survives staff turnover
Lessons outlast the grant and the people who learned them.
- Why
- Teams relearn the same lessons when staff leave or a grant ends.
- How
- A program keeps what it tried, why, what it learned, what remains uncertain, and why it stopped anything. The organization keeps its own copy of what it learned, and the copy remains the organization’s if it stops working with Prova.
- How we will know
- The next person, team, or community can understand or do more because of what survived.
Findings that change what happens
Organizations change course when they learn they could help more.
- Why
- A finding changes people’s lives only when the organization can act on it.
- How
- Leaders get an explanation they can examine and use: what the evidence supports, what remains uncertain, and which choices are still theirs. The reasons for a decision are kept, so a later team can see why it was made.
- How we will know
- The organization responds, and its response changes something for the people it serves. An average can hide an unfair burden, so improvement is judged from the lives of the people affected.
Answerable to the people served
The people affected have a say while decisions can still change.
- Why
- The people living with a program often see first when it is failing them. An employment program can place people in jobs quickly and still miss whether the work is safe or fits their family responsibilities.
- How
- Questions and findings are written so that people can engage with them, and what they say can change the question. The report says what people can influence and who decides. Privacy and the freedom to decline are respected.
- How we will know
- Affected people gain a practical ability to shape or challenge something that matters.
Shared evidence across organizations
Organizations work together and keep what makes each of them different.
- Why
- Many important questions fall between agencies and programs. A public agency knows the rules, a nonprofit knows delivery, and a community group hears what people experience.
- How
- A shared report shows what each partner’s evidence supports and where the partners disagree. Responsibilities are clear, each partner decides what it shares, and smaller partners do not carry all the reporting.
- How we will know
- Worthwhile work becomes possible together, and the relationship stays fair to everyone in it.
Early tests for new ideas
A promising idea can be tried before it is proven.
- Why
- Many ideas from the people closest to a need never get a test, because a study costs more than the idea has raised.
- How
- Prova helps clarify an idea, find what is already known, and name what needs testing. Early results are reported as early results, so a small trial does not turn into a promise to expand. Changing or stopping the idea is a sound result.
- How we will know
- People can understand or responsibly try options they could not before, including learning that an attractive idea should not go ahead.
Three things have to go right.
Much of this depends on things nobody can know in advance. Three of them matter most.
- The cost has to fall far enough for an ordinary program to afford a credible answer.
- Funders, agencies, and the public have to rely on the checks when they decide.
- The checks have to stay honest as they get cheaper, and answer to the people they describe.
No single method settles how much a program helps.
A careful comparison can show an effect but miss why it happened, or for whom. The experience of the people a program serves can show what no study measured, but it may be partial. Good records can show a pattern, but they may hold only what was easy to record. Good judgment comes from setting these side by side.
The people a program serves often see first when it is failing them, and why. Their explanation belongs in the evidence, and they should be able to read and question what is said about them.
Of 90 education programs tested in randomized trials commissioned by the Institute of Education Sciences from 2002 to 2013, 79 showed weak or no positive effects.20
Cheaper evidence brings its own risks.
When counting becomes easy, what gets counted tends to be what is easiest to count. A number can be quoted long after the caveats that came with it have been dropped. A finding from one place can be applied in another place where it does not hold. Evidence about people can be used against them, or gathered without their consent. Once a measure carries money, the work can start to bend toward moving the measure, and then it stops showing what it once showed.21 Lower costs leave all of these risks in place, and some of them grow as evidence becomes easier to produce.
“The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
The aim is an honest report of what is known, how confidently, by what means, and what remains open. A stranger can rely on a report like that, and a community can question it.
If you would like to support this work, question it, or build on it, we would like to hear from you.
Start with Prova →Notes
- Federal grant spending. Congressional Research Service, Federal Grants to State and Local Governments: Trends and Issues, R40638, updated June 26, 2025, summary. CRS estimates that federal grant outlays to state and local governments were $1.1 trillion in fiscal year 2024, across health care, transportation, education, income security, and other areas. The figure measures the volume of funding. It does not show how much of that funding is supported by evidence.
- The “less than $1 of every $100” estimate. John Bridgeland and Peter Orszag, “Can Government Play Moneyball?,” The Atlantic, July/August 2013; an excerpt is published by Results for America. The authors describe the estimate as based on rough calculations. It is their 2013 estimate, and it is not an audited measure. Its scope and period differ from the 2024 grant total in note 1, so the two figures cannot be combined into an amount of unevaluated spending.
- Nonprofit evaluation budgets and staffing. Innovation Network, State of Evaluation 2016: Evaluation Practice and Capacity in the Nonprofit Sector, 2016, printed pages 1, 3, 5, and 8. The survey received responses from 1,125 US nonprofit organizations. The budget finding covers the 740 organizations that answered that question: 84 percent spent less than 5 percent of their organizational budget on evaluation. The staffing finding covers the 786 organizations that evaluated their work: 6 percent named internal evaluation staff as primarily responsible for evaluation, and 2 percent named an external evaluator. The report combines these as 8 percent having evaluation staff, internal or external.
- Audited financial statements under securities law. Congressional Research Service, Auditing and Accounting Regulation: Key SEC Powers, RS21257, July 8, 2002, printed page 3. The report explains that federal securities law requires the financial statements of registered companies to be audited by independent public or certified accountants. “Companies selling securities to the public” refers to this registration framework, which has its own requirements and exemptions. The account of how investment was allocated before independent audits, and of how it widened afterward, is Prova’s own summary of that history.
- The 1962 drug-effectiveness requirement. Michelle Meadows, “Promoting Safe & Effective Drugs for 100 Years,” US Food and Drug Administration, FDA Consumer, January–February 2006. The article explains that the Kefauver–Harris Drug Amendments of 1962 required firms seeking to market a drug to provide substantial evidence of its effectiveness for the intended use, based on adequate and well-controlled studies.
- The 1948 streptomycin trial. Medical Research Council, “Streptomycin Treatment of Pulmonary Tuberculosis: A Medical Research Council Investigation,” BMJ 2 (1948): 769–782, documented by the James Lind Library, which links to the original article. The Library describes the trial as a methodological landmark, particularly for its detailed procedures to keep treatment assignments from being known in advance. Earlier controlled trials exist; this one is cited as a landmark.
- The National Bureau of Standards. Arden Bement, “For the 2nd USA-China Seminar on Technical Innovation,” National Institute of Standards and Technology, March 19, 2002. Bement describes how Congress established the National Bureau of Standards in 1901 in response to inconsistent measurement standards, and its later work in materials testing, calibration, and reference materials. Applying this history to checking claims about social programs is an analogy.
- Federal managers’ access to evaluations. US Government Accountability Office, Program Evaluation: Strategies to Facilitate Agencies’ Use of Evaluation in Program Management and Policy Making, GAO-13-570, June 26, 2013, highlights and appendix I. GAO surveyed a stratified random sample of 4,391 federal managers and supervisors and received 2,762 usable questionnaires, between November 2012 and February 2013. In its governmentwide estimates, 37 percent reported that a program, operation, or project they were involved in had been evaluated within the previous five years, and 40 percent did not know whether one had been. The percentages describe what managers reported and knew. They cannot be read as shares of federal programs or spending.
- The evidence commission’s assessment. Commission on Evidence-Based Policymaking, The Promise of Evidence-Based Policymaking, September 2017, printed page 1. The quoted words appear in the executive summary, where “this need” refers to policymakers’ need for evidence to improve government programs and policies. The statement is the bipartisan commission’s assessment in 2017.
- The 2016 economics prize. Nobel Prize Outreach, “The Prize in Economic Sciences 2016.” The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel was awarded jointly to Oliver Hart and Bengt Holmström for their contributions to contract theory. The Nobel committee’s explanatory account discusses performance measurement, multitasking, incomplete contracts, and the allocation of decision rights. This work examines how contracts and incentives depend on what can be observed, measured, or specified.
- Incentives across multiple tasks. Bengt Holmström and Paul Milgrom, “Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design,” Journal of Law, Economics, and Organization 7, special issue (1991): 24–52, especially pages 25 and 26; accessible article text. The paper models how rewarding one part of a job can draw attention away from the others. Its examples include rewarding teachers for measured student performance, and rewarding production volume when quality or the upkeep of equipment is harder to measure. The model describes how rewards interact with measurement and competing tasks. It does not say that measurement always distorts work, or that stronger incentives are always better.
- The 2019 economics prize. Nobel Prize Outreach, “The Prize in Economic Sciences 2019.” The prize was awarded jointly to Abhijit Banerjee, Esther Duflo, and Michael Kremer for their experimental approach to alleviating global poverty. The official explanation describes how that approach developed and what it contributed. Connecting this research with contract theory to argue for an affordable institution that tests and checks claims about programs is Prova’s own interpretation and founding thesis.
- Nonprofit CEOs’ intentions to leave. Ben Gose, “A Really Tough Job: Nonprofit CEOs Work to Make Their Roles More Manageable,” Chronicle of Philanthropy, May 7, 2024. More than 350 nonprofit chief executives took part in a survey commissioned by the Chronicle, and one-third said they were likely to leave their organization within two years. The article does not give the dates of the survey. The finding reports what chief executives intended. It does not measure departures, or the knowledge lost when someone leaves.
- Support during leadership transitions. Frances Kunreuther, Sean Thomas-Breitfeld, and Tessa Constantine, The Push and Pull: Declining Interest in Nonprofit Leadership, Building Movement Project, 2024, page 13, figure 9. Drawing on the 2022 Race to Lead survey, the figure reports that 22 percent of responding executive directors of color and 30 percent of responding white executive directors received support from the previous executive director during their transition. The figures report support during the transition. They do not measure whether records were kept or how much knowledge was handed over.
- Evaluation budgets in the Social Innovation Fund. Lily Zandniapour and Nicole Vicinanza, Budgeting for Rigorous Evaluation: Insights from the Social Innovation Fund, Corporation for National and Community Service, October 2013, pages 4 and 9 and tables 4 and 5 on pages 23 and 24. Among 70 evaluation plans, 15 aimed at a strong level of evidence. Those 15 had a mean annual evaluation budget of $589,037 and a median of $379,800. Their evaluation budgets averaged 28 percent of the program budget, with a median of 24 percent and a range of 11 to 83 percent. These are budgets from one federal grant portfolio. The percentages compare evaluation with the program’s budget, while note 3 compares it with the whole organization’s budget. Page 4 cites guidance of 5 to 7 percent, says the rule of thumb has risen to about 10 percent, and cautions that a fixed percentage may not be enough for a rigorous evaluation. The account of how long credible studies took, who paid for them, and how rarely their results could be compared is Prova’s own reading of the field.
- MDRC’s research history. MDRC, “About MDRC,” accessed November 1, 2026. MDRC was founded in 1974 and conducts rigorous studies of programs and policies that affect people with low incomes. The statement that most of the methods a credible study needs have been settled for decades is Prova’s own reading of the field, with MDRC’s history as one illustration. Each study still needs its own design and judgment.
- J-PAL’s evaluation count. Abdul Latif Jameel Poverty Action Lab, “About Us,” accessed November 1, 2026. J-PAL reports more than 2,700 completed or ongoing randomized evaluations in more than 140 countries. The count includes work still under way.
- AI-assisted data extraction. Gerald Gartlehner and colleagues, “Artificial Intelligence–Assisted Data Extraction With a Large Language Model: A Study Within Reviews,” Annals of Internal Medicine 178, no. 12 (2025): 1763–1771, DOI: 10.7326/ANNALS-25-00739. The study compared 9,341 data elements from 63 studies within six ongoing systematic reviews. Claude models made the first extraction, and a person checked it. Accuracy was 91.0 percent for the AI-assisted process and 89.0 percent for people working alone, with a median time saving of 41 minutes per study. Accuracy was measured for each data element, including whether the study reported it at all. The authors name the subjective judgment involved in classifying errors, and the difficulty of tracking time consistently, as limitations. The study covers one task, data extraction. The wider statement that language models can now take on part of the reading, source tracing, and checking, with a person checking the result, is Prova’s own assessment.
- The Evidence Act. Foundations for Evidence-Based Policymaking Act of 2018, Public Law 115-435, enacted January 14, 2019, section 101, adding 5 U.S.C. §§ 311–313. Section 311 defines the agencies covered by reference to 31 U.S.C. § 901(b). Section 312 requires an evidence-building plan within each agency’s strategic plan, and an annual evaluation plan. Section 313 requires each agency to name a senior employee as its Evaluation Officer. “Federal agencies” and “each agency” refer to these covered agencies. The Act sets planning and organizational duties; it does not require an impact evaluation of every federally funded program. The CRS summary gives an overview.
- The Coalition’s review of IES-commissioned trials. Coalition for Evidence-Based Policy, Randomized Controlled Trials Commissioned by the Institute of Education Sciences Since 2002: How Many Found Positive Versus Weak or No Effects, July 2013, pages 1 and 2. The Coalition classified 79 of 90 educational interventions as producing weak or no positive effects compared with usual school practice. The 90 are interventions, and some trials tested more than one. The classification gave priority to final, policy-relevant outcomes and to longer follow-up. It counted effects that were not statistically significant as weak, along with two significant effects judged small, at 0.05 and 0.06 standard deviations. The classification is the Coalition’s own, made in 2013 from the research then available. Weak or no positive effects is a different finding from zero effect, and it does not describe social programs in general.
- Campbell’s warning about consequential indicators. Donald T. Campbell, “Assessing the Impact of Planned Social Change,” Evaluation and Program Planning 2, no. 1 (1979): 67–90. The quoted passage also appears on printed page 34 of an accessible reprint of an earlier version, published in the Journal of MultiDisciplinary Evaluation 7, no. 15 (2011). The 1979 date is that of the journal publication; Campbell stated the warning earlier. He describes how indicators that carry consequences for decisions come under pressure to be corrupted, and to distort the processes they monitor, and he calls his illustrative evidence mostly anecdotal. The surrounding discussion of privacy, consent, and applying findings in new places reflects Prova’s own judgment.