Have this sheet beside you when you read a report, a dashboard or a reply to your request. Find the one number the decision rests on and work down the questions. You will not be able to answer all of them. Each unanswered question is something you can ask the institution.
Before you open the figures, write one sentence about what you expect them to show, and one about what would surprise you. With that written down first, you can tell afterward whether the figures changed your reading or only confirmed it.
01What is being counted?
- What exactly is the unit? A person, a household, a visit, a trip, a case, a complaint? One person can make many visits; one household can hold several people.
- What is the written definition? "Served," "resolved," "vacant" and "in need" each depend on a rule someone wrote. Ask for the rule.
- How was it collected? A machine count, a staff tally, a form, a survey, an estimate? Each misses different things.
- Is it a count or a rate? A count tells you how many. A rate (per hour, per 1,000 residents, per dollar) tells you how many compared with something. Know which one you have, and what it is divided by.
Ask for: the definition, the collection method, and the figures before they were turned into a rate.
02Who is counted, and who is missed?
- Who is included? Which people, places or services fall inside the count?
- Who would not show up? People who avoid the counter, the form or the office; people who use a service in a way the method does not record; people whose language is not on the form; people who gave up before they were counted.
- Were some figures estimated or filled in? Look for footnotes, asterisks, "estimated," "imputed" or "provisional."
- Are small numbers hidden? Institutions often suppress or round small counts to protect privacy. That is usually appropriate, but it can make small neighborhoods or groups disappear from a table.
Ask for: a note of who the method cannot see, and which figures were estimated and how.
03Since when?
- What period does it cover? A month, a year, a school term? Is that period typical, or did something unusual happen in it?
- Did the definition or method change during the period? A new form, a new system or a new rule can move a number without anything on the ground changing.
- Is it the latest available? Find the date the figure was produced as well as the period it covers.
Ask for: the same figure for earlier periods, with a note of any change in how it was counted.
04Compared with what?
- What is the comparison? Other neighborhoods, last year, a target, a standard, the average? A number alone rarely supports a decision; the comparison does the work.
- Is the comparison like for like? Places with different sizes, populations, opening hours or needs may need a rate rather than a count, or a different rate.
- What would the number look like for the alternative? If the decision chose one option over another, was the same figure produced for both?
- Is the difference large enough to matter? Small differences in small numbers can come and go from year to year by chance. Look at more than one period before treating a ranking as fixed.
Ask for: the figures for the comparison group and for the option that was not chosen.
05Before you draw a conclusion
- Keep "not shown" separate from "not true." If the number cannot show what you care about, that is a gap in the evidence, not proof of the opposite.
- Check that the number matches the stated reason. A decision about one group of residents needs figures for that group; a decision about a neighborhood needs neighborhood figures, not a citywide average.
- Ask someone outside the group to read it. If they understand the number differently, the number needs a clearer definition, or you do.
Related: Asking an Institution for the Evidence Behind a Decision, for the reasons behind asking; A Completed Request and Reply, Annotated, for a worked example.
06Further reading
- 1Joel Best, Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists (University of California Press, 2001; updated edition 2012). How social statistics are produced, and the questions that expose a misleading one.
- 2David Spiegelhalter, The Art of Statistics: Learning from Data (Pelican, 2019; Basic Books, 2019). Clear chapters on counting, rates, comparisons and chance variation.
- 3Tim Harford, The Data Detective: Ten Easy Rules to Make Sense of Statistics (Riverhead, 2021). Practical rules for reading a number in context, including asking what is missing.
- 4U.S. Census Bureau, Understanding and Using American Community Survey Data: What All Data Users Need to Know (2020). How estimates, margins of error and comparisons work in the most common source of US neighborhood data.