Example
Northline’s draft says the job-readiness program increased lasting employment by 30% for the people they serve. The intake log, the follow-up tracker, and the field definitions are already on the shared drive. This page follows that sentence through those files, while the wording can still be changed.
Synthetic organization · illustrative example
Northline runs a job-readiness program at two sites. 214 people completed last year. An Ashgrove Foundation renewal is approaching, and the board has asked for evidence of sustained employment outcomes.
2 sites · 214 completions · $3.1M annual budget
Ahead of the renewalWhat they already had
The draft annual report says the program increased lasting employment by 30%. The comparison behind that number is people who enquired and never enrolled. The same folder has an intake log that offers places in referral-date order until the 60 are filled.
The sentence
Our job-readiness program increased lasting employment by 30% for the people we serve.
A board will remember a sentence like this. The comparison behind it is people who enquired and never enrolled.
In the files already
Places are capped at 60. Later referrals wait.
Places are offered in the order people were referred, until the 60 are filled. The 87 who wait look like the 60 who got in. What separates them is that the places ran out. That comparison is fairer than people who enquired and never enrolled.
Cohort capacity: 60 places per intake.
Applicants are ordered by referral date. Places are offered in that order until capacity is reached.
Applicants above the cut are enrolled. Applicants below the cut are held to the next intake.
The consideration this raised
The methods by which a credible comparison condition is constructed when randomization is not feasible.
What the record supports
The sentence type fits, but its population, measure, source, or analytic status needs to travel with it.
What this comparison can support
Quasi-experimentalfrom a cutoff, waitlist, capacity rule, boundary, or phased rolloutThe files
Report
page 4 · circulated to the board 2 April
Program results
Our job-readiness program increased lasting employment by 30% for the people we serve.
Across two sites, 214 people completed the program in the reporting year.
We are grateful to the Ashgrove Foundation for three years of support.
What the check returned
The selected record can carry a quasi-experimental statement, subject to the qualifications below.
The report compares people who enrolled with people who enquired and then stayed away. Those two groups may already have been different in ways that affect whether they find work. The intake log offers the 60 places in referral-date order. The people held over look like the people who got in.
The waitlist is already in the operating files. Nobody had to build a comparison for this check.
What if they change one thing
The comparison gets stronger, and that finding holds. How many people were reached, and what employed means, still have to sit with the sentence. Those limits are the next part of the check.
What the record can carry
Within the selected operational comparison, employment was 30% higher than among people who did not enrol among program completers. This can support a causal interpretation of the job-readiness program only if the design's identifying assumptions hold. The selected measure is a proxy for the outcome named in the claim. Outcomes were visible for 42% of the relevant group.
The smallest useful next step
Report the observed and missing groups separately, then test how good and bad outcomes among the missing group would change the conclusion.
Applied because the files and the decision made them relevant.
The same filesThe 90 people
Follow-up reached 90 of 214 people. The field records any paid work in the 30 days before the call. A funder is being asked to rely on that number. A narrower sentence is one the board can be shown.
What the record supports
The sentence type fits, but its population, measure, source, or analytic status needs to travel with it.
What this comparison can support
Quasi-experimentalfrom a cutoff, waitlist, capacity rule, boundary, or phased rolloutThe files
Data export
program database · exported 28 March
Completed program: 214
Reached at six months: 90
Follow-up rate: 42%
Not reached: 124 — no working number (71), no response after three attempts (38), declined (15)
Outreach calls were made Tuesday and Thursday, 9am–4pm.
What the check returned
Follow-up covers 42% of the relevant group; the missing outcomes could materially change the reported pattern.
The 30% describes the 90 people who answered the six-month call. The outcomes of the other 124 are unknown, and there are enough of them to move the number either way.
The 90 and the 124 are already written down, with reasons.
If people who are hardest to reach also have different outcomes, the observed group may not represent the full group.
Outreach ran on weekday daytimes. Someone who has started work is harder to reach on a Tuesday morning, so the people missing from the number may be missing for a reason connected to the outcome being measured.
The tracker already says calls went out Tuesday and Thursday, 9 to 4.
The selected measure is a proxy for the outcome named in the claim.
The report says lasting employment. The field records any paid work in the 30 days before the call. The data supports the second statement, and the report makes the first.
The database defines employed as any paid work in those 30 days. Hours, continuity, and pay were never in the field.
For a consequential decision, a program-controlled measure benefits from an independent cross-check.
The organization being funded also collected and controls the outcome record, and a renewal decision depends on it. On a decision of that weight, a second source held outside the program is worth having.
The outcome file is still Northline's. The renewal has not happened yet.
What if they change one thing
The comparison holds. How many people were reached, what employed means, and who holds the numbers still sit with the sentence.
Work that can be dropped
Northline had budgeted a fresh six-month outcome survey to strengthen the renewal case. It would ask the same 30-day question of the same reachable group. A larger sample would leave the sentence where it is, three months later.
The consideration this raised
The matching of design rigor to decision stakes and value of information, so that the intensity of investigation is calibrated to what resolving the uncertainty is worth rather than defaulting to the most rigorous design the budget allows.
Where this leaves them
The waitlist already in the intake log is a fairer comparison than the one in the draft. The sentence still has to name the 90 people reached, the 30-day measure, and the remaining limits. The expansion question needs a different check.
What the record can carry
Within the selected operational comparison, employment was 30% higher than among people who did not enrol among program completers. This can support a causal interpretation of the job-readiness program only if the design's identifying assumptions hold. The selected measure is a proxy for the outcome named in the claim. Outcomes were visible for 42% of the relevant group.
The smallest useful next step
Report the observed and missing groups separately, then test how good and bad outcomes among the missing group would change the conclusion.
Applied because the files and the decision made them relevant.
Six weeks laterThe expansion question
The same files now have to answer a larger question. Staging the rollout and opening one new site leave the answer where it was. Filling the coach posts moves it.
What the record supports
Workforce, training, inputs, quality measurement, or founder-led delivery will break before the proposed jump. The pilot result is not a delivery system.
The files
Proposal
FY27 · tabled 6 April
Recommendation
Extend the job-readiness program to Eastvale and Rill Street in FY27.
Target: 2× current annual enrolment.
New sites will report the same monthly indicators as existing sites.
What the check returned
Delivery risk is high. The triggering facts are: constrained critical inputs.
The result was produced at a 1:20 caseload. The new sites are budgeted at 1:32, and four of eleven coach posts in the region have stood vacant for more than six months. The model in the proposal differs from the one the result came from.
The staffing plan already shows the 1:20 caseload, the 1:32 budget at new sites, and the vacant coach posts.
Transport risk is medium. Watch: mixed population match; mixed site match; uptake that is likely lower or unknown; heterogeneity not tested.
The new sites draw from a different referral stream, with longer unemployment durations. Whether the program works the same way for that group is an open question, and this expansion is the first thing that makes it worth testing.
The draft referral agreement names the longer unemployment and less stable housing before the sites open.
System risk is medium. Watch: a possible or present shared market; possible or unknown displacement; material or unknown spillovers; fixed partner capacity; changing or unknown policy.
Four sites competing for placements with the same eleven employers is a different situation from two. Some of the gain at the new sites could be placements moved from the existing ones.
The working note already says eleven employers account for 62% of placements, and all four sites would share them.
Learning risk is medium. Watch: no credible comparison at the next stage.
The new sites will report the same monthly indicators, which describe activity. Establishing whether the expansion worked would require a comparison the plan has yet to include.
The proposal already says the new sites will report the same monthly numbers.
A randomized, multi-site, or credible comparison was reported. That does not lower transport or system risk. It only says the original estimate may be real under pilot conditions.
The waitlist finding from the first check is still in the files. The original estimate may well be real under the conditions that produced it, and what it establishes is confined to those conditions.
The first check stays. The expansion question leaves that finding in place.
Founder-led delivery or extra support is a support factor. Later sites will not automatically inherit it.
The employer relationships that make placements happen are held personally by the program lead. A fourth site would need to build its own.
The staffing plan already names who holds the employer relationships.
What if they change one thing
The coach posts, the shared employers, and how they would know if it worked still carry cautions.
What to do before the vote
Secure the constrained input, or redesign delivery around what the ordinary supply can hold.
This result is a structured decision aid based on the answers you selected. It does not verify the underlying facts, estimate a treatment effect at scale, certify readiness, or replace a professionally owned expansion judgment. No result here means the program is ready for full scale.
Applied because the files and the decision made them relevant.
Your own decision
The Decision Brief asks what is being decided, who needs the answer, by when, what information already exists, and where the uncertainty sits. It runs in your browser, and it reaches Prova only if you choose to send it.
Prepare a decision briefNorthline is a synthetic organization, and this example is illustrative. It carries no conclusion about any real program or client.