The Office for Statistics Regulation wrote to Sai about Methodology of Justice Data Lab statistics

19 February 2025 · OSR, Office for Statistics Regulation

The letter is dated 19 February 2025. The Office for Statistics Regulation published it as OSR to Sai: Methodology of Justice Data Lab statistics; it links to 5 official records. Below are 4 passages from the letter in the regulator's own words.

What the Office for Statistics Regulation wrote

“I respectfully request that you review the methodology of data published by the MoJ Justice Data Lab, for a. failure to conduct multiple comparisons correction (MCC) / account for family-wise error (FWE), and b. possible p-hacking by failure to disclose all considered comparisons (e.g. ones that were unfavourable).”

“I request that you please review MoJ’s statistical methods, publications, and underlying data, and — should you agree that they are faulty — require them to retract any published claims of statistical significance that are not actually justified and publish corrected statistics (including all considered but unpublished values).”

“We recognise that multiple comparison correction (MCC) is widely considered to be best practice when carrying out a large number of comparisons on the same data.”

“In examining these statistics, we have identified some improvements that we consider MoJ could make to the presentation and communication of the statistics.”

From OSR, Office for Statistics Regulation, letter of 19 February 2025. Read the full letter.

The words the letter quotes

The letter quotes 3 passages from the public record:

“While multiple correction methods can be applied to reduce the risk of incorrectly finding a positive treatment effect, they can also increase the likelihood that real differences will not be detected.”

Source named in the letter: 2023 TSP reoffending report

“In examining these statistics, we have identified some improvements that we consider MoJ could make to the presentation and communication of the statistics.”

“We are happy to keep you updated on this work.”

Records the letter links to

• https://www.gov.uk/government/statistics/justice-data-lab-statistics-april-2024 (gov.uk)
• https://assets.publishing.service.gov.uk/media/66602fffdc15efdddf1a8756/HMPPS_CFO_JDL_Report.pdf (assets.publishing.service.gov.uk)
• https://assets.publishing.service.gov.uk/media/6628f8683b0122a378a7e5a0/Justice_Data_Lab_general_annex.pdf (assets.publishing.service.gov.uk)
• https://www.gov.uk/government/collections/justice-data-lab-pilot-statistics (gov.uk)
• 2023 TSP reoffending report

Receipt