Psychological Measurement: What Does the Score Mean?

Free MCAT content · Psychology · Research

Separate the attribute researchers want to study from the score they collect, then evaluate consistency, validity, bias and causal limits.

About 12 minutes, including the worked example and checks.

Check your reasoning · Continue learning · Sources

Concept: a construct needs an observable measure

A psychological construct is the attribute a researcher wants to study, such as persistence, perceived control or working-memory capacity. An operational definition specifies how the study will record that attribute. Naming the construct does not yet specify a measurement.¹

For academic persistence, one study might count seconds spent on a difficult optional task. Another might total responses to questions about continuing after setbacks. Those procedures produce different kinds of evidence. The first records behavior in a particular setting; the second records a person’s report about their behavior. Neither automatically represents every aspect of persistence.

Keep three statements separate: what the researcher means by the construct, what participants actually do, and how the resulting observations become a score. That separation makes it easier to see where an interpretation can fail.

Pattern: consistency and meaning require different evidence

Reliability concerns consistency. Test-retest evidence compares the same measure across occasions. Interrater evidence compares observers evaluating the same material. Internal consistency examines how items intended to contribute to a measure relate to one another. Each addresses a different source of variation.²

Validity concerns the evidence supporting the intended interpretation of the score. A reproducible score can still reflect the wrong attribute. For example, a difficult reading task might consistently rank people by vocabulary while its author labels the score “persistence.” Consistency alone cannot justify that label.²

Useful validity evidence can include adequate coverage of the construct, expected relationships with other measures, and relationships with a relevant behavioral criterion. Face validity is the surface impression that a measure looks appropriate. That impression is a starting point, not a substitute for empirical evidence.

Interpret reliability in context. A stable characteristic should usually produce reasonably stable scores under comparable conditions. A measure of a changing state, such as momentary frustration, can change because the person changed. A low correlation across widely separated occasions does not identify measurement error by itself.

Rule: connect each finding to the claim it supports

When a study reports that a measure “works,” locate the actual observation. Repeated scores address temporal consistency. Agreement between observers addresses their use of a scoring rule. Association with a different relevant measure addresses whether the proposed interpretation behaves as expected. One kind of evidence cannot automatically stand in for the others.

AAMC research-design reasoning includes evaluating the appropriateness, reliability and validity of behavioral measures, the implications of self-report, and the difference between an association and a causal claim.³

Application: a questionnaire about persistence

Consider an invented study of 80 student volunteers. Researchers define academic persistence as continuing effort on a chosen academic task after difficulty arises. Their new questionnaire contains eight statements, each rated from 0 to 4; the summed score ranges from 0 to 32.

The same students complete the questionnaire again seven days later under similar conditions, with no planned intervention. They also attempt a challenging optional puzzle task. The behavioral measure is the number of seconds spent actively working before choosing to stop, up to a 12-minute limit. Recordings allow two observers to apply a shared definition of active work.

All findings in this example are illustrative. Questionnaire scores correlate at r = 0.84 across the two occasions. Initial scores correlate at r = 0.41 with time spent working on the puzzle task. No uncertainty intervals are supplied.

What these findings contribute

The first relationship supports relative stability of questionnaire scores over this interval. It does not establish that every student’s score stayed identical. The second is evidence consistent with a relationship between the questionnaire and a relevant behavior. It does not establish that the questionnaire measures only persistence or works equally well in every population.

The behavioral task also needs scrutiny. Puzzle skill, interest and perceived difficulty could affect time spent working. The 12-minute cap prevents the task from distinguishing participants who would have continued for different lengths of time beyond that limit. An observer’s judgment about “active work” introduces another measurement decision. Treating a behavioral measure as objective does not remove these interpretation problems.

A useful next study could test whether the questionnaire relates to persistence across several kinds of tasks while examining plausible alternative explanations. Observers could also score the same recordings independently to establish whether the scoring instructions yield consistent judgments.

Response bias can be systematic and repeatable

A participant may answer in a way that appears responsible or acceptable rather than reporting their usual behavior. This is socially desirable responding. Cues about the researcher’s expected result can also shape responses. Both can affect a measure even when its scores repeat consistently.⁴

In the persistence example, “I always keep working until every task is complete” invites a different judgment from a concrete question about a specified recent task. Broad wording, unclear time frames and the response choices can change how participants retrieve and report information. Survey construction therefore affects what a score means.⁵

Private responses, neutral instructions, clear time frames and consistent administration can reduce avoidable influences. They do not guarantee unbiased answers. An independent behavioral measure supplies another perspective, although its own limits still require evaluation.

Association supports prediction before it supports causation

Suppose questionnaire scores also predict how long students persist on a later assignment. That would add evidence about a future criterion. It would not show that increasing the questionnaire score causes students to persist longer. Motivation, prior success or another variable could influence both, and the study has not assigned an intervention that isolates the proposed cause.⁶

The defensible statement identifies the measurement, relationship and setting: higher questionnaire scores were associated with greater observed persistence in these volunteers. A stronger causal or population-wide statement requires additional evidence addressing that specific claim.

Check your reasoning

Open an answer to see the reasoning. Each check stands on its own.

Apply it

In the illustrative persistence study, two observers independently score the same puzzle recordings. Their active-work estimates differ substantially, even though questionnaire scores are stable across the two administrations. Which concern is most directly supported?

A. Behavioral scoring has weak interrater reliability under the current procedure.

Correct.

The observers rated the same recordings and disagreed. That directly challenges consistency across raters. The separate questionnaire can still show temporal stability.

B. Participants changed their persistence substantially between questionnaire sessions.

Review this choice.

The disagreement occurs between observers of the same behavior. It does not establish a change in the participants across time.

C. Questionnaire scores have weak test-retest reliability over seven days.

Review this choice.

The prompt describes stable questionnaire scores. Observer disagreement concerns a different measure and a different kind of reliability.

D. Self-reported persistence is inflated by socially desirable responding.

Review this choice.

Social desirability remains a possible concern for self-report, but disagreement between behavioral observers does not demonstrate it.

The principle to carry forward

Locate the source of disagreement before naming the problem. Different observers, different items and different occasions test different forms of consistency.

Try a different application

Changed case: researchers develop a questionnaire about confidence in handling academic difficulty. They predict that higher scores will be associated with greater persistence on challenging tasks. Scores predict persistence on a new task six weeks later. The questionnaire is administered only once, and no intervention is assigned. Which conclusion has the most direct support?

A. Changing self-confidence would cause a change in later persistence.

Review this choice.

The questionnaire precedes the task, but temporal order alone does not isolate cause. Other variables could influence both scores and later persistence.

B. Consistency of the questionnaire across occasions has been established.

Review this choice.

The questionnaire was administered once. A later behavioral observation is a different measure, so it does not establish test-retest reliability.

C. Prediction of a later relevant behavior adds criterion-related validity evidence.

Correct.

The score relates to a later behavior that matters to its proposed interpretation. This adds predictive criterion-related evidence in the studied setting without establishing causation or complete validity.

D. Response bias has been excluded by measuring the outcome at a later time.

Review this choice.

A later outcome does not remove possible bias from the original self-report. A biased measure can still correlate with another variable.

The principle to carry forward

Prediction, repeated measurement and causal manipulation answer different questions. Match the strength of the conclusion to the evidence actually collected.