Transparency Lab
What the official data actually says
This section holds the difficult, derived and fragmented public data: figures that exist only in a workbook footnote, only after joining two official datasets, only across many authorities, or not at all. Everything here is reproducible from first-party sources. Where we calculate a figure ourselves it is labelled a Civometry calculation and shows its numerator, denominator, formula and vintage; it never looks like a government-published figure.
The product standard
Every statistic on this site can be followed along one trail:
- Map
- area
- statistic
- raw count
- rate
- numerator
- denominator
- formula
- official dataset
- exact source
- methodology
- limitations
Don't trust us. Check us.
Every controversial statistic resolves to one of four outcomes:
- the official figure
- a reproducible calculation from official figures
- the data is incomplete and here is what is missing
- the government does not publish enough to answer
We never claim that a figure is withheld on purpose. We classify how accessible it is and say what is missing.
Areas
- Ethnicity and criminal justiceIn progress
Exact criminal-justice measures by ethnicity (stop and search, arrests, prosecutions, convictions, custody, prison population, use of force, reoffending) from Home Office and Ministry of Justice tables, with rates per 100,000 calculated against Census 2021 ethnic-group populations and every numerator, denominator and vintage shown.
- Ethnicity and the criminal justice system (in progress)
- Stop and search by ethnicity (official table)
- Prison population by ethnicity (official table)
- Data recipes for each derived rate (in progress)
- 12 data gaps
- Religion and demographic changePlanned
Census 2011 and 2021 religion and ethnic-group counts for every local authority, MSOA and LSOA, from the ONS Nomis API, with change shown in people, in percentage terms and in percentage points of share, and the absence of any official religion series between censuses stated.
- Religion in England and Wales (planned)
- Population topic hub
- 10 data gaps
- Places of worshipPlanned
Planning applications for places of worship read from the MHCLG Planning Data platform and local planning authority registers, classified only from the official proposal text, with application, permission, refusal and confirmed completion kept separate and the authority coverage shown.
- Places of worship (planned)
- 3 data gaps
- PlanningPlanned
Planning application records by authority from planning.data.gov.uk, with the number of authorities supplying data, the share of England covered and the latest update per authority, so that no figure is extrapolated to a national total.
- Planning applications by authority (planned)
- 1 data gap
- Government data gapsLive
Every question this site has been asked of official data, classified by what is actually published: available, derivable, partial, fragmented, FOI required, not published, not collected, restricted for disclosure, or unknown, with what exists, what is missing and which authority holds it.
- FOI-derived datasetsIn progress
Datasets assembled from Freedom of Information responses where no publication exists, with each request, authority, deadline, response and released file recorded, and coverage shown by authority so that a partial set of responses is never presented as a national figure.
- Cost of living and household budgetsIn progress
Household spending averages by household type, income decile and region from the ONS Living Costs and Food Survey, local rents from the Price Index of Private Rents, Band D council tax by billing authority and average house prices, assembled into Civometry estimates that state the geographic level of every input and are labelled as estimates, never as official figures about any household.
- What does it cost to live here? (Civometry estimate)
- Cost of living by local authority (official figures)
- Where does our income sit? (official deciles)
- How much does a child cost? (official spending gap)
- What is my personal inflation rate? (Civometry calculation)
- How much will energy cost? (DESNZ consumption at the Ofgem cap)
- 5 data gaps
Source accessibility levels
Each registered source carries a level from 1 to 7 describing how its figures reach us. The level says nothing about why a publisher chose a format. Levels 4 and above describe figures rather than files: a level 4 figure is one we calculate from two or more official datasets and document as a data recipe; levels 5 to 7 are recorded in the data-gaps register, not against a source.
| Level | Meaning | Example | Sources |
|---|---|---|---|
| 1 | Official API. Published through an official programmatic interface that returns the figures directly, with codes and periods. | ONS Nomis API for Census 2021 tables | 22 |
| 2 | Official machine-readable download. Published as CSV, JSON or a zip of such files that can be read without locating cells in a layout. | data.police.uk monthly archive | 4 |
| 3 | Spreadsheet requiring extraction. Published in a workbook (XLSX or ODS) where the figure must be located by sheet, table, row label and column header. | ONS Crime in England and Wales appendix tables; Home Office and Ministry of Justice ODS tables | 71 |
| 4 | Multiple official datasets joined. No single publication gives the figure; it is calculated by joining two or more official datasets (for example a count and a population). | Recorded crime per 1,000 residents by local authority | — |
| 5 | Fragmented across authorities. Held by many separate authorities in different formats with no national compilation, so coverage must be tracked per authority. | Planning applications for places of worship | — |
| 6 | FOI required. Held by a public authority but not published; obtainable, if at all, through a Freedom of Information request. | Requests recorded on /foi | — |
| 7 | Not published. Not published by any authority, and in some cases not collected. | Ethnicity of the offender for police-recorded offences with no identified suspect | — |
Level 1 sources (22)
- ASHE resident analysis, 2002 onwards (ONS via Nomis, NM_30_1)ONS · CSV · Open, documented, machine-readable API returning csv; no key needed at this volume. L1 · Official API
- Census 2021 TS001 - Number of usual residents in households and communal establishments (Nomis)ONS · APIL1 · Official API
- Census 2021 TS021 - Ethnic group (Nomis)ONS · APIL1 · Official API
- Census 2021 TS030 - Religion (Nomis)ONS · APIL1 · Official API
- Census 2021 RM032 - Ethnic group by sex by age (Nomis)ONS · APIL1 · Official API
- Census 2011 KS101EW - Usual resident population (Nomis)ONS · APIL1 · Official API
- Census 2011 KS201EW - Ethnic group (Nomis)ONS · APIL1 · Official API
- Census 2011 KS209EW - Religion (Nomis)ONS · APIL1 · Official API
- Census 2021 TS004 - Country of birth (Nomis)ONS · APIL1 · Official API
- Census 2011 KS204EW - Country of birth (Nomis)ONS · APIL1 · Official API
- Census 2021 custom dataset - Tenure by country of birth of Household Reference PersonONS · APIL1 · Official API
- Census 2021 custom dataset - Economic activity by country of birthONS · APIL1 · Official API
- Census 2021 custom dataset - Highest qualification by country of birthONS · APIL1 · Official API
- Census 2021 custom dataset - NS-SEC by country of birthONS · APIL1 · Official API
- Census 2021 custom dataset - Tenure by ethnic group of Household Reference PersonONS · APIL1 · Official API
- Explore Education Statistics: absence, SEN, attainment, apprenticeships, NEET, class sizesDepartment for Education · API · Open, documented JSON API with stable ids for every data set, filter option and indicator, and a query endpoint that returns the published cells directly. No key, no scraping. L1 · Official API
- ONS Open Geography Portal generalised boundaries (BGC/BSC), WGS84 GeoJSONONS · JSONL1 · Official API
- Claimant count by sex and age (Nomis)ONS · APIL1 · Official API
- Labour market overview, UK (LMS time series)ONS · API · Open JSON time series API, no key, one URL per series, with the release date, unit and title carried in the response. L1 · Official API
- Consumer price inflation time series (MM23)ONS · API · Open JSON time series API, no key, one URL per series, with the release date, unit and title carried in the response. L1 · Official API
- Public sector finances time series (PUSF)ONS · API · Open JSON time series API, no key, one URL per series, with the release date, unit and title carried in the response. L1 · Official API
- National accounts time series (PN2 and UKEA)ONS · API · Open JSON time series API, no key, one URL per series, with the release date, unit and title carried in the response. L1 · Official API
Level 2 sources (4)
- data.police.uk street-level crime, cumulative monthly archiveHome Office · ZIP · Monthly zip of per-force CSV files; machine-readable as downloaded. Turning it into local rates needs the ONS LSOA lookup and a population denominator, which is a level 4 derivation described on /transparency. L2 · Official machine-readable download
- National Statistics Postcode Lookup (NSPL), quarterly editionONS · CSVL2 · Official machine-readable download
- ONS Open Geography Portal names, codes and best-fit lookups (constituencies, regions, countries, MSOA, LSOA, wards)ONS · CSVL2 · Official machine-readable download
- UK House Price Index: average pricesHM Land Registry · CSV · Open CSV at a predictable monthly URL, one row per area and month, with ONS area codes. No key, no scraping, no manual step. L2 · Official machine-readable download
Level 3 sources (71)
- Long-term international migration, provisionalONS · XLSXL3 · Spreadsheet requiring extraction
- Crime in England and Wales: Appendix tablesONS · XLSXL3 · Spreadsheet requiring extraction
- Long-term international migration 2.00, citizenship, UK (1964 to 2019)ONS · XLSX · Discontinued fixed .xls file; last edition November 2020. L3 · Spreadsheet requiring extraction
- Historical crime data: recorded crime 1898 to 2001/02Home Office · XLSX · Fixed legacy workbook; series definitions change across the 1898-2002 span and are documented only in footnotes. L3 · Spreadsheet requiring extraction
- Immigration statistics, year ending March 2019: asylum tables volume 1 (as_01 to as_02)Home Office · ODSL3 · Spreadsheet requiring extraction
- Immigration statistics, year ending March 2019: entry clearance visa tables volume 1 (vi_01 to vi_05_q)Home Office · ODSL3 · Spreadsheet requiring extraction
- Offender management statistics quarterly: annual prison population tables, 2019 edition (June 2002 to June 2019)Ministry of Justice · ODSL3 · Spreadsheet requiring extraction
- Offender management statistics quarterly: annual prison population tablesMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Council tax levels set by local authorities in England, area tables 2020-21 to 2024-25 (MHCLG)MHCLG · ODS · Open ODS and XLSX at each year's statistical release page. One file per year, a different asset URL each time, and the table number moves between editions. L3 · Spreadsheet requiring extraction
- Police powers and procedures: arrestsHome Office · ODS · Arrests by self-defined ethnicity are published as counts (A.01c) and, for England and Wales, England and Wales as a rate per 1,000 population against the 2021 Census (A.04). Rates by ethnicity for each police force area appear only in a separate "ethnic disparity time series" file; rates per 100,000 and any age-standardised rate are not published. L3 · Spreadsheet requiring extraction
- Police use of force statistics, England and WalesHome Office · ODS · Use of force reports by self-defined or officer-perceived ethnicity are published as counts (Table 4a) and as the Home Office's own rate per 1,000 population against the 2021 Census (Table 15). One report is one person against whom force was used in one incident; a person can appear in several reports. L3 · Spreadsheet requiring extraction
- Ethnicity and the criminal justice system: offender management tablesMinistry of Justice · ODS · Biennial compendium (the cadence enum has no "biennial"; the next edition is due November 2027). Table 6.02 gives the average sentence length of determinate-sentence prisoners released in each calendar year, by self-identified ethnicity. Prosecutions, convictions and custody rates by ethnicity are not in the 2024 edition's tables; they exist only inside the Criminal Justice System statistics quarterly "outcomes by offence" data tool (published only as raw data). L3 · Spreadsheet requiring extraction
- Police force area data tablesONS · XLSXL3 · Spreadsheet requiring extraction
- Band D council tax rates by year, 1996-97 onwards (Scottish Government)Scottish Government · XLSXL3 · Spreadsheet requiring extraction
- Recorded Crime in Scotland: bulletin tablesScottish Government · XLSXL3 · Spreadsheet requiring extraction
- Police Recorded Crime in Northern Ireland: annual tablesPolice Service of Northern Ireland · XLSXL3 · Spreadsheet requiring extraction
- Benefit expenditure and caseload tables: outturn and forecast (Spring Forecast 2026)DWP · XLSX · Open xlsx on GOV.UK, consistently laid out, but with a new asset URL each edition and forecast years mixed into the same rows as outturn. L3 · Spreadsheet requiring extraction
- Regional and local authority gas consumption statistics (DESNZ)DESNZ · XLSX · Open XLSX on the statistical release page, one sheet per year with a clean header row; the asset URL changes each edition. L3 · Spreadsheet requiring extraction
- Regional and local authority electricity consumption statistics (DESNZ)DESNZ · XLSX · Open XLSX on the statistical release page, one sheet per year with a clean header row; the asset URL changes each edition. L3 · Spreadsheet requiring extraction
- Estimates of the population for England and Wales (mid-year), local authority tablesONS · XLSX · One workbook per edition; every prior year is revised in each edition, so the vintage must be tracked by hand. L3 · Spreadsheet requiring extraction
- Mid-year population estimates time series (National Records of Scotland)National Records of Scotland · XLSX · One workbook holding the whole back-run; each edition revises earlier years, so the vintage must be tracked by hand. L3 · Spreadsheet requiring extraction
- Mid-year population estimates for Northern Ireland, administrative area totals (NISRA)NISRA · XLSX · One workbook per edition; the sheet is a tidy flat table, but the file path carries the edition month. L3 · Spreadsheet requiring extraction
- HMRC tax and NICs receipts for the UKHMRC · ODS · Open ODS spreadsheet at a stable publication page, well structured with one row per financial year and one column per tax, but the asset URL changes with each edition so the file has to be resolved from the page. L3 · Spreadsheet requiring extraction
- Immigration system statistics: asylum claims and initial decisions datasetsHome Office · XLSX · Long-format (one row per combination) XLSX; tidy, but still a workbook with cover and notes sheets rather than a CSV or API. L3 · Spreadsheet requiring extraction
- Immigration system statistics: asylum claims awaiting a decision datasetHome Office · XLSX · Long-format XLSX with as-at date strings rather than ISO dates. L3 · Spreadsheet requiring extraction
- Immigration system statistics: asylum seekers in receipt of support datasetHome Office · XLSXL3 · Spreadsheet requiring extraction
- Immigration system statistics: asylum seekers in receipt of support by local authority datasetHome Office · XLSX · Long-format XLSX; local authority names, not ONS codes, so a name-to-code match is needed. L3 · Spreadsheet requiring extraction
- Immigration system statistics: illegal entry routes detailed datasetHome Office · XLSXL3 · Spreadsheet requiring extraction
- Immigration system statistics: immigration detention detailed datasetsHome Office · XLSXL3 · Spreadsheet requiring extraction
- Immigration system statistics: returns detailed datasetsHome Office · XLSXL3 · Spreadsheet requiring extraction
- Immigration system statistics: entry clearance visa outcomes datasetsHome Office · XLSXL3 · Spreadsheet requiring extraction
- Immigration system statistics: asylum summary tablesHome Office · ODSL3 · Spreadsheet requiring extraction
- Immigration system statistics: entry clearance visas summary tablesHome Office · ODSL3 · Spreadsheet requiring extraction
- Immigration system statistics: safe and legal (humanitarian) routes summary tablesHome Office · ODSL3 · Spreadsheet requiring extraction
- Immigration system statistics: illegal entry routes summary tablesHome Office · ODSL3 · Spreadsheet requiring extraction
- Immigration system statistics: immigration detention summary tablesHome Office · ODSL3 · Spreadsheet requiring extraction
- Immigration system statistics: returns summary tablesHome Office · ODSL3 · Spreadsheet requiring extraction
- Migrants detected crossing the English Channel in small boats: time seriesHome Office · ODS · Weekly ODS on a publication page whose asset path changes each release. L3 · Spreadsheet requiring extraction
- ASHE Table 8.7a: annual gross pay by place of residence (ONS)ONS · XLSX · Zip of xlsx workbooks on the ONS dataset page, one workbook per measure, one sheet per employee group, one row per area with ONS codes. The 8.7a workbook is fed to this family.L3 · Spreadsheet requiring extraction
- ASHE Table 8.1a: weekly gross pay by place of residence (ONS)ONS · XLSX · Same zip as the annual table; the 8.1a workbook is fed to this family.L3 · Spreadsheet requiring extraction
- Price Index of Private Rents, UK: monthly price statistics (ONS)ONS · XLSX · One xlsx on the ONS dataset page, one tidy sheet with a row per month and area and ONS area codes; the asset path changes each month.L3 · Spreadsheet requiring extraction
- Family Spending workbooks: expenditure by income, region and household type (ONS)ONS · XLSX · Workbook 2 (by income) and workbook 3 (by region) are separate xlsx files on the ONS dataset pages, with multi-row headers and footnote markers; each is loaded as its own release of this family.L3 · Spreadsheet requiring extraction
- English Indices of Deprivation: local authority district summariesMHCLG · XLSX · An open workbook on GOV.UK, one sheet per domain and one row per district, found among eleven files on the release page rather than at a stable data URL. A new index replaces the file wholesale every few years. L3 · Spreadsheet requiring extraction
- Prison population: weekly estate figuresMinistry of Justice · ODS · One ODS per week on a yearly collection page; the asset path changes each week. L3 · Spreadsheet requiring extraction
- Criminal court statistics quarterlyMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Crime outcomes in England and WalesHome Office · ODSL3 · Spreadsheet requiring extraction
- Homicide in England and Wales: appendix tablesONS · XLSXL3 · Spreadsheet requiring extraction
- Offender management statistics quarterly: prison populationMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Offender management statistics quarterly: probationMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Prison population: monthly bulletinMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Criminal justice statistics quarterly: overview tablesMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Proven reoffending statistics quarterlyMinistry of Justice · ODSL3 · Spreadsheet requiring extraction
- Police workforce, England and WalesHome Office · ODSL3 · Spreadsheet requiring extraction
- Police powers and procedures: stop and searchHome Office · ODSL3 · Spreadsheet requiring extraction
- Components of population change by local authority (ONS mid-year estimates, sheet MYE3)ONS · XLSX · One workbook per edition holding one year only. A back-run means reading each edition in turn, and each edition revises the years before it, so the vintage must be tracked by hand. L3 · Spreadsheet requiring extraction
- Components of population change, mid-2011 to mid-2024 back series (ONS MYEB, sheet MYEB3)ONS · XLSX · One workbook holds the whole back run for every UK local authority, so no vintage-chasing is needed — but each edition rewrites the entire history, so this source must be re-ingested whole when a new edition lands, never spliced with the previous one. L3 · Spreadsheet requiring extraction
- Consultant-led referral to treatment waiting times: overview timeseriesNHS England · XLSX · A workbook on england.nhs.uk whose URL carries a random token that changes with every monthly release, behind bot protection that refuses scripted requests. Finding it means navigating the statistics pages in a browser; nothing about the data itself is open to automation. L3 · Spreadsheet requiring extraction
- A&E attendances and emergency admissions: monthly timeseriesNHS England · XLS · A legacy .xls workbook on england.nhs.uk whose URL carries a random token that changes with every monthly release, behind bot protection that refuses scripted requests. Acquired through a browser each month. L3 · Spreadsheet requiring extraction
- Historical official forecasts databaseOBR · XLSX · Open XLSX on obr.uk with one sheet per variable and one row per forecast vintage, but the site returns a bot-verification challenge to scripted requests, so each release must be saved from a browser by hand before it can be ingested. L3 · Spreadsheet requiring extraction
- Public finances databankOBR · XLSX · Open XLSX on obr.uk, one row per financial year; the site returns a bot-verification challenge to scripted requests, so each release must be saved from a browser by hand before it can be ingested. L3 · Spreadsheet requiring extraction
- Average annual domestic electricity bills, UK (DESNZ table 2.2.1)DESNZ · XLSX · Open XLSX at a stable statistical data set page, one row per year, but the asset URL changes with each edition and the workbook mixes calculation sheets with the published table. L3 · Spreadsheet requiring extraction
- Average annual domestic gas bills, UK (DESNZ table 2.3.1)DESNZ · XLSX · Open XLSX at a stable statistical data set page, one row per year with payment-method columns; the asset URL changes with each edition. L3 · Spreadsheet requiring extraction
- Weekly road fuel prices, UK (DESNZ)DESNZ · XLSX · Open XLSX and CSV at a stable page, one row per week since June 2003, updated every Tuesday; the asset URL changes each week. L3 · Spreadsheet requiring extraction
- Licensed vehicles at the end of the quarter, UK (DfT VEH0101)Department for Transport · ODS · Open ODS at a stable statistical data set page, tidy layout with Geography and Date columns; the asset URL changes with each edition. L3 · Spreadsheet requiring extraction
- Reported road collisions, casualties and vehicles involved, Great Britain (DfT RAS0101)Department for Transport · ODS · Open ODS at a stable data-tables page, one row per year from 1926, two tables side by side on one sheet; the asset URL changes with each edition. L3 · Spreadsheet requiring extraction
- Council tax levels set by local authorities in England 2026 to 2027 (MHCLG)MHCLG · ODS · Open ODS at the annual statistical release page, small clean tables, asset URL changes each year. L3 · Spreadsheet requiring extraction
- Public Expenditure Statistical Analyses 2025, chapter 4: expenditure by functionHM Treasury · XLSX · Open XLSX at the annual PESA publication page, one workbook per chapter with one row per function and financial years across the columns; the asset URL changes each year and footnote markers sit inside the row labels. L3 · Spreadsheet requiring extraction
- Public Expenditure Statistical Analyses 2025, chapter 5: expenditure by sub-functionHM Treasury · XLSX · Open XLSX at the annual PESA publication page; table 5.2 has one row per COFOG sub-function and five financial years across the columns, with footnote markers inside some row labels. L3 · Spreadsheet requiring extraction
- Police workforce open data table, England and Wales (Home Office)Home Office · ODS · One ODS holds the whole 2007-to-date panel, republished whole each July with the workforce release under a new asset URL. L3 · Spreadsheet requiring extraction
- Population estimates for the UK and constituent countries: historical time seriesONS · XLSX · Open XLSX on ons.gov.uk with one row per mid-year estimate; the file path changes with each edition's year range. L3 · Spreadsheet requiring extraction
- Social housing lettings in England, Tenants Summary Tables (MHCLG CORE)MHCLG · ODS · Open ODS on each year's statistical release page, with a different asset URL each edition. The nationality time series (Table 3e) is republished whole every year, so one file carries the run. L3 · Spreadsheet requiring extraction
Local recorded-crime rates per 1,000 residents are a level 4 figure: data.police.uk street-level counts (level 2) joined to the ONS mid-year population estimates (level 3) through the ONS LSOA lookup. The population year and the coverage of every force are shown on each local page.
Data gaps
47 questions are classified in the data-gaps register:
| Status | Questions |
|---|---|
| Available | 10 |
| Derivable | 4 |
| Partial | 11 |
| Fragmented | 7 |
| FOI required | 2 |
| Not published | 5 |
| Not collected | 4 |
| Disclosure restricted | 3 |
| Unknown | 1 |
Freedom of Information
Where a figure is held but not published, we ask for it and record the request, the deadline, the response and every released file. No requests have been sent yet; the candidates are listed on the FOI page. See /foi.
Challenges and corrections
Any statistic can be challenged on methodological grounds: the official source, the numerator, the denominator, the geographic or category mapping, the period, the formula or the interpretation. Accepted challenges produce a public correction that records the original value, the corrected value, the affected geography and the reason, and the original is never silently replaced. 1 correction published. See /transparency/challenges.