Statistical significance: what it means
A test of whether a difference between estimates is larger than would be expected from sampling variation alone. It does not mean the difference is large or important.
A difference is statistically significant when a statistical test indicates it is unlikely to be the result of random sampling variation alone. Official bulletins commonly use a 5% significance level, linked to 95% confidence intervals.
Worked example
A hypothetical survey shows 20% of people experienced an issue last year and 21% this year. With a large sample, a one-point rise might be significant; with a small sample, the same rise might not be. Significance depends on the size of the change and the sample, not only on the change.
What it does and does not mean
- Significant does not mean large, important or caused by a particular policy.
- Not significant does not prove there was no change; the data may be too limited to detect it.
- Testing many comparisons will produce some significant results by chance.
How it trips up claims
- A claim that something "rose" based on a non-significant change in a survey goes beyond what the producer reports. The Crime Survey for England and Wales, for example, states whether each change is statistically significant.
- Significance tests apply to sample data, not to complete counts such as registered deaths.
Also searched as: statistically significant, significance test, p-value, what does statistically significant mean
Key figures where it matters
Questions that use this term
Related terms
- Confidence interval: A range around an estimate that shows how much it could vary because it comes from a sample. A 95% interval is the usual standard.
- Survey sample: The group of people or businesses selected to take part in a survey, chosen so that their answers can be used to estimate figures for the whole population.
- Odds, risk, relative and absolute risk: Risk is the chance an event happens out of everyone exposed; odds compare those it happens to with those it does not. Relative risk compares two groups' risks.
More statistics, surveys and quality terms: Accredited official statistics · Official statistics in development · Pre-release access · Survey sample · Response rate · Administrative data · Provisional figures · Revisions · Confidence interval · Headline figure · Sample size · Labour Force Survey (LFS).