Statistical significance and effect size: what a result means

MCAT content library / Research & data / Statistical significance and effect size: what a result means

Evaluate the estimated effect, its uncertainty, and the study design before interpreting a significance label.

About 16 minutes, including the worked examples and checks.

From a dataset to a supported claim · Lesson 5 of 6 · Continue in this path

Useful foundations

• Confidence intervals, error bars, and sample size
• Observational and experimental studies: what can we conclude?

On this page

• Concept: separate the size of a result from its compatibility with a model
• Pattern: the same estimated effect, different uncertainty
• Rule: ask three different questions
• Application: compare the estimate with a useful-effect threshold
• Distinguish a threshold decision from a sharp scientific boundary
• Check your reasoning
• Continue learning
• Sources

Concept: separate the size of a result from its compatibility with a model

A study can estimate a small difference very precisely or a large difference imprecisely. A significance label cannot tell you which occurred. Start with the estimated effect in its units, then examine its uncertainty and the study design. Only after that should you interpret a reported p-value.

A p-value describes how extreme the observed test statistic is relative to a specified model. For a two-sided test, it includes results at least as extreme in either relevant direction, as defined by that test. A small value can indicate tension between the data and the null model together with its assumptions. It is not the probability that the null hypothesis is true.¹

The null model must be identified. In a comparison of means, it might specify a zero population difference. Other tests address other parameter values or relationships. The test statistic, the assumed distribution, the sampling design, and the analysis decisions all contribute to what the p-value means.

Pattern: the same estimated effect, different uncertainty

The following original studies compare a training method with a control method using a score measured in points. Both estimate a +2-point mean difference. They have different numbers of independent units and different reported standard errors. For this illustrative comparison, use the stated two-sided normal-approximation test and approximate 95% intervals calculated as estimate ±1.96 ×SE.

Study	Units per group	Estimated difference	SE of difference	Two-sided p-value	Approximate 95% CI
A	40	+2 points	2 points	0.317	−1.92 to 5.92 points
B	640	+2 points	0.5 points	0.000063	1.02 to 2.98 points

Against a zero-difference null, Study A’s standardized statistic is 2/2 = 1. Study B’s is 2/0.5 = 4. Under the specified standard-normal reference, a statistic at least four units from zero is much less common than one at least one unit from zero. The estimated effect has not become larger; it is estimated more precisely in B.

Using a prespecified 0.05 threshold, A does not cross the threshold and B does. This does not mean A proves no effect while B proves a large effect. A’s interval includes a range of negative and positive differences. B’s narrower interval supports a small positive difference under its model.

Rule: ask three different questions

What was estimated? Here it was a mean difference of 2 points. Report the scale and direction. A relative change, a percentage-point change, a correlation, and a difference in means describe different quantities. A standardized effect divides a difference by a specified scale of variability; it is still not a p-value.

How uncertain is the estimate? The interval in B excludes zero but is close to zero compared with some possible practical thresholds. The interval in A is wider and includes zero. Neither interval assigns a frequency to individual outcomes. A comparison of means can be informative while individual responses overlap extensively.

What can this design establish? A small p-value does not repair confounding, selective follow-up, an inappropriate comparison, or a biased outcome measure. If participants chose their training methods, a precise association can still reflect baseline differences. Random assignment and appropriate conduct support a different causal argument from self-selection.²

Application: compare the estimate with a useful-effect threshold

Suppose the research team specified in advance that an improvement of at least 5 points would justify replacing a costly training program. This is a decision criterion for this fictional setting, not a general rule for educational research.

Study B’s interval of 1.02 to 2.98 lies below 5. Its small p-value against zero does not show that the useful-effect criterion has been met. Study A’s interval reaches 5.92 but also includes negative differences. A therefore remains uncertain about a 5-point improvement; the upper endpoint is not evidence that such an improvement has been demonstrated.

If the team’s question is whether the method achieves at least 5 points, that question should be reflected in the estimand and analysis. A zero-effect test and a useful-effect assessment answer different questions. Likewise, a failure to reject zero is not an equivalence test: demonstrating that effects lie within a narrow acceptable range requires an appropriate design and analysis.

Distinguish a threshold decision from a sharp scientific boundary

A prespecified significance threshold can define a testing rule. Under its null model and assumptions, the procedure controls the long-run false-positive rate at the chosen level. That does not make 0.049 and 0.051 fundamentally different descriptions of the world. Report the estimates, intervals and analysis context rather than replacing them with a yes/no label.¹

The same issue arises when comparing two studies. One result crossing a threshold and another not crossing it does not establish that their effects differ. In the table, the point estimates are identical. To support a claim that two effects differ, analyze the relevant contrast between those effects with its uncertainty.

Repeated analysis choices also matter. Trying many outcomes, subgroups, transformations, or stopping points and presenting only the most favorable p-value changes the evidence available to a reader. A meaningful interpretation needs the analysis plan and a transparent account of what was examined. Multiplicity may require an appropriate adjustment or a clearly exploratory interpretation.³

For an MCAT-style data question, keep the claim proportional to the information supplied. If the study reports a difference and interval, use them. If it reports only a threshold label, do not invent an effect magnitude. If the design is observational, retain the causal limitation even when the numerical evidence against the stated null is strong. AAMC’s statistical-reasoning scope asks students to combine data and limitations when evaluating conclusions.⁴

Check your reasoning

Choose an answer before opening its feedback. Each option explains the reasoning behind that choice.

Continue learning

From a dataset to a supported claim

Lesson 5 of 6.

• Previous in From a dataset to a supported claim: Confidence intervals, error bars, and sample size
• Next in From a dataset to a supported claim: Reading a multi-figure experiment from claim to conclusion
• Return to this path’s overview

See every step in From a dataset to a supported claim

From a dataset to a supported claim

Interpret variability and uncertainty before combining evidence from multiple figures.

1. Reading graphs: axes, scales, and the comparison that matters
2. Mean, median, and spread: describing a dataset
3. Accuracy, precision, and sources of measurement error
4. Confidence intervals, error bars, and sample size
5. Statistical significance and effect size: what the result means · You are here
6. Reading a multi-figure experiment from claim to conclusion

Related lessons

• Confidence intervals, error bars, and sample size
• Observational and experimental studies: what can we conclude?
• Reading a multi-figure experiment from claim to conclusion

Browse research & data · MCAT content library

MCAT preparation and roadmap · Aim High, Score Higher

Sources

1. Wasserstein and Lazar: The ASA Statement on p-Values

2. AAMC: Reasoning about Design and Execution of Research

3. American Statistical Association: Statement release and six principles

4. AAMC: Data-based Statistical Reasoning