MCAT content library / Research & data / Sampling and generalizability: trace who enters the estimate
Identify who could enter a study, who responded, and which population the resulting estimate can reasonably describe.
About 14 minutes, including the worked examples and checks.
Useful foundations
On this page
Concept: who could enter the study?
A sample can be large and still tell you little about the population named in a conclusion. Begin by distinguishing the target population, the sampling frame used to reach that population, the people invited, and the people who actually provide data. A loss at any transition can matter if the excluded people differ on the outcome. Probability sampling uses a random selection process with known selection probabilities; those probabilities need not be equal. A convenience sample recruits accessible people without that probability structure.
For example, a campus email directory may cover currently enrolled students but omit students on leave. A daytime booth reaches people who pass that location. An open survey link reaches people who see it and choose to respond. None of these details tells you that a particular estimate must be wrong. They tell you which assumptions are needed before extending the estimate beyond its respondents. Statistics Canada distinguishes a sample’s selection method from the broader population it is intended to describe.1
Pattern: follow selection separately from measurement
Imagine a study of whether students can attend a free review session. The target is every student enrolled this semester. Investigators invite a random sample from a complete enrollment list, but the questionnaire is available only during a narrow afternoon window. Students working at that time may be less likely to respond and less able to attend the proposed session. Random invitations protect the initial selection step; they do not force the final respondents to resemble everyone invited.
This is a possible nonresponse bias: response is related to the quantity being estimated. A low response rate raises a question about bias, but the rate alone does not establish its direction or magnitude. If respondents and nonrespondents have similar attendance constraints, a low response rate could coexist with little bias. Conversely, a high response rate can still miss a small group whose circumstances are important to the decision. Compare available information about respondents, nonrespondents, and the frame instead of treating a percentage as a verdict.2
Keep a second distinction in view. Selection determines whose experiences are represented; measurement determines what is recorded about those experiences. A perfectly worded attendance question cannot restore omitted evening students. Recruiting evening students cannot repair a question that confuses willingness to attend with ability to attend. In a passage, identify the threatened step before choosing a design correction.
Application: a pooled result and a weighted estimate
All numbers and study scenarios on this page are original teaching examples. Suppose enrollment records show that 60% of students have daytime schedules and 40% have evening schedules. A random invitation sample happens to include 120 daytime and 80 evening students.
| Schedule | Invited | Responded | Respondents able to attend |
|---|---|---|---|
| Daytime | 120 | 96 | 72 |
| Evening | 80 | 16 | 4 |
The response rates are 96/120 = 80% and 16/80 = 20%. Among respondents, the attendance proportions are 72/96 = 75% and 4/16 = 25%. The pooled respondent proportion is (72 + 4)/(96 + 16) = 76/112, or approximately 67.9%. That arithmetic accurately describes the respondents. It does not automatically describe enrollment.
Using the known schedule proportions gives a different estimate: (0.60 × 0.75) + (0.40 × 0.25) = 0.55, or 55%. This calculation represents each schedule group according to its enrollment share rather than its share of respondents. The difference is understandable: daytime students, who report higher availability, make up about 86% of respondents but only 60% of enrollment.
The weighted estimate still needs an assumption: within each schedule group, respondents must adequately represent the attendance availability of the students whose responses are missing. If evening students with the longest work shifts are especially unlikely to respond, the 25% evening estimate may remain too high. Weighting the measured schedule variable cannot recover information about every unmeasured selection process. Treat 55% as the estimate under the stated within-group assumption, not as a demonstrated population truth.
Rule: match the repair to the missing people
A useful correction changes the process producing the imbalance. Offer several response windows, contact a random subset of initial nonrespondents through a feasible alternative mode, and compare their answers with the initial responses. These steps can reveal whether the observed schedule imbalance also accompanies an outcome difference. Simply collecting more responses through the same daytime window may make the estimate more precise around the same biased value.
Distinguish a probability sample from a sample that merely looks balanced. A volunteer sample containing equal numbers of daytime and evening students still depends on who volunteered within those groups. Conversely, a probability design may deliberately sample a smaller group at a higher rate and use the known selection probabilities during estimation. The visible group counts alone do not identify the design. The selection process does.3
Keep sampling and assignment in separate sentences
Random sampling concerns entry into a study. Random assignment concerns allocation of enrolled participants to conditions. If 80 volunteers are randomly assigned to two teaching formats, the comparison can support a causal interpretation for the trial participants under the usual design and follow-up assumptions. It does not make those volunteers a probability sample of every student. Motivation, schedule flexibility, and prior preparation could affect whether the same effect would occur elsewhere.5
Before accepting a population claim, name the population, the recruitment route, the important missing groups, and the outcome-related reason they might differ. Then state the narrowest supported conclusion. “Among the respondents, 67.9% reported availability” is directly descriptive. “About 55% of all enrolled students are available” requires the weighting assumptions. “Students at every university would respond similarly” requires additional evidence about settings the study never sampled.
ACTIVE PRACTICE · SAMPLING AND SELECTION
Distinguish the Respondent Rate From an Enrollment Estimate
The source experiment invites 120 daytime and 80 evening students. Daytime: 96 respond and 72 report availability; evening: 16 respond and 4 report availability. Enrollment is 60% daytime and 40% evening. Response may be selective even after the original random invitations.
| Schedule | Invited | Responded | Available | Within-group availability |
|---|---|---|---|---|
| Daytime | 120 | 96 | 72 | 75% |
| Evening | 80 | 16 | 4 | 25% |
Raw respondent proportion: 76/112 = 67.9%. Schedule-weighted respondent estimate: 0.60(0.75) + 0.40(0.25) = 55.0%.
Illustrative population availability with daytime fixed at 75%: . Difference from the 55% respondent-based weighted estimate:.
The sensitivity slider is a hypothetical assumption, not recovered missing responses. Daytime true availability is also unknown; holding it at 75% isolates one source of possible bias.
Check your reasoning
Choose an answer before opening its feedback. Each option explains the reasoning behind that choice.
Check 1
In the example, what assumption is specifically needed to interpret the weighted 55% as an estimate for all enrolled students?
A. Daytime and evening students have equal attendance availability.
Review this choice. The weighting calculation allows the groups to have different availability: 75% and 25%. Equality is neither assumed nor observed.
B. Initial invitations contain equal numbers from each schedule group.
Review this choice. The calculation uses known enrollment shares; equal invitation counts are not required for the stated estimate.
C. Respondents represent attendance availability within each schedule group.
Correct. The weights correct the group mixture. They do not correct outcome-related nonresponse that remains within a schedule group.
D. Response rates are equal after the two groups are combined.
Review this choice. Combining groups yields one overall response rate, but that does not remove the unequal response processes.
The principle to carry forward
Separate correcting the mixture of measured groups from correcting selection within each group.
Check 2
A second campus survey uses a list that excludes students on leave, although the target includes them. Which change most directly addresses this particular problem?
A. Add students on leave to the frame before drawing the sample.
Correct. The identified problem is frame undercoverage. Giving the omitted target members a chance of selection directly addresses it.
B. Increase the number sampled from the current list.
Review this choice. A larger sample from an incomplete list still gives the omitted group no chance of entry.
C. Randomly assign current respondents to two questionnaire orders.
Review this choice. Questionnaire order might affect measurement, but it cannot include people absent from the recruitment frame.
D. Use the same question wording in every invitation.
Review this choice. Consistent wording addresses comparability of measurement, not missing target-population members.
The principle to carry forward
A design correction should act at the stage where the identified bias enters.
Check 3
A trial recruits highly motivated volunteers and randomly assigns them to two study schedules. Retention and measurement are comparable between groups. Which statement best separates the two design questions?
A. Random assignment makes the volunteers representative of all examinees.
Review this choice. Assignment balances the allocation process among those enrolled; it does not change who volunteered.
B. Random assignment supports the treatment contrast, while broader generalization needs additional evidence.
Correct. The randomized contrast and the extension to a broader population depend on different parts of the design.
C. Random sampling is necessary before any comparison of assigned groups can be informative.
Review this choice. An informative randomized contrast can exist within a selected sample, even when external validity is limited.
D. Volunteer recruitment prevents the assigned schedules from being compared within the trial.
Review this choice. Volunteer recruitment limits some population claims but does not erase the information from random allocation within the trial.
The principle to carry forward
Ask how people entered the study and how conditions were assigned as two separate questions.
Continue learning
Related lessons
- Observational and experimental studies: what can we conclude?
- Surveys, interviews, and ethnography: choose evidence for the question
Browse research & data · MCAT content library
MCAT preparation and roadmap · Aim High, Score Higher
Sources
1. Statistics Canada — Selection of a sample
2. Statistics Canada — Questions to ask about surveys
3. Statistics Canada — Probability sampling