Some of the trickiest Problem-Solving and Data Analysis questions don't involve any calculation at all — they ask you to judge whether a study's conclusion is actually supported by its design. This guide explains the difference between observational studies and experiments, when causation claims are valid, and works through two full examples.
Why the SAT Tests Study Design, Not Just Numbers
Evaluating statistical claims is its own official subtopic within Problem-Solving and Data Analysis, and it tests something distinct from arithmetic: whether you can reason about how a study was conducted and whether its conclusion — often about cause and effect — is actually justified by that design. Getting these questions right means reading carefully for a handful of specific words: "random," "assigned," "observed," "sample," and "population."
Observational Studies vs. Experiments
In an observational study, researchers watch and record what subjects are already doing, without assigning them to different conditions. In an experiment, researchers actively and (ideally) randomly assign subjects to different treatment groups and compare outcomes.
This distinction matters enormously for what conclusions are valid:
- Observational studies can show that two variables are associated, but cannot establish that one causes the other, because other unmeasured factors (confounding variables) might explain the pattern.
- Well-designed randomized experiments can support causal claims, because random assignment spreads confounding factors evenly across groups, isolating the effect of the treatment itself.
Worked Example: Judging a Study's Conclusion
Consider two versions of a study on the same question:
| Study | Design | Result |
|---|---|---|
| Study 1 | Researchers surveyed 500 adults and recorded their reported coffee consumption and sleep quality. | Adults who reported drinking more coffee also reported lower sleep quality. |
| Study 2 | Researchers randomly assigned 500 volunteers to either drink coffee daily or avoid it, then measured sleep quality after four weeks. | The coffee group had measurably lower sleep quality than the no-coffee group. |
Question: Which study, if either, supports the conclusion that "drinking coffee causes lower sleep quality," and why?
Step 1 — Classify each study's design. Study 1 only observed existing behavior (an observational study); Study 2 randomly assigned subjects to groups (an experiment).
Step 2 — Apply the rule. Study 1 can only support an association between coffee consumption and sleep quality — it's possible that something else, like stress levels or work schedule, independently affects both coffee drinking and sleep. Study 2's random assignment means the coffee and no-coffee groups should be similar in every other respect on average, so the sleep quality difference can be reasonably attributed to coffee itself.
Conclusion: Only Study 2 supports a causal claim. Study 1 supports only the weaker claim that coffee consumption and sleep quality are associated.
This exact "which study allows a cause-and-effect conclusion" question format appears repeatedly, and the answer almost always comes down to spotting the word "randomly assigned."
Worked Example: Distinguishing Random Assignment from Random Selection
These two ideas get confused constantly, so here's a second scenario that isolates the difference:
| Study | How the sample was chosen | How treatment was given |
|---|---|---|
| Study 3 | Researchers randomly selected 300 tomato plants from a large commercial farm's entire crop. | Plants were then randomly assigned to receive either a new fertilizer or the standard one. |
| Study 4 | Researchers used 300 tomato plants volunteered by a gardening club, not randomly selected from a larger population. | Plants were randomly assigned to receive either the new fertilizer or the standard one. |
Question: In which study, if either, can the conclusion "the new fertilizer causes higher yield" be safely generalized to all tomato plants of that type, not just the ones tested?
Step 1 — Check random assignment (needed for causation) in both. Both Study 3 and Study 4 randomly assigned plants to treatment groups, so both can support a causal claim about the plants actually studied.
Step 2 — Check random selection into the sample (needed for generalizing beyond the study). Only Study 3 randomly selected its plants from the farm's full crop; Study 4 used a self-selected, volunteered group from a gardening club, which may not represent tomato plants in general.
Conclusion: Study 3 supports both a causal claim and generalizing that claim broadly. Study 4 supports a causal claim only for the specific plants in that study — its result can't be confidently extended beyond the gardening club's plants without further evidence.
This is the exact two-part check the SAT rewards: random assignment answers "can I claim cause and effect?" while random selection answers "can I extend that claim beyond this sample?" A study can have one without the other, and the strongest possible design has both.
Generalizing Results to a Population
A separate but related question type asks whether a study's results can be generalized beyond its sample. This depends on how the sample was selected, not how the treatment was assigned. If Study 2's 500 volunteers were not randomly selected from a broader population — say, they were all college students who volunteered — then the results might only reasonably generalize to people similar to that group, not to all adults. Random assignment supports causal claims within the study; random selection supports generalizing findings beyond the study.
Common Wrong-Answer Patterns
- Claiming causation from an observational study — always the most common trap answer choice.
- Assuming a small randomized experiment generalizes broadly without random selection into the sample in the first place.
- Ignoring confounding variables that could explain an association without any causal link at all.
- Confusing random assignment with random selection — a study can have one without the other, and each supports a different type of conclusion.
Frequently asked questions
Does a large sample size make an observational study support causation?
No. Sample size affects precision (how confident you can be in the association), but it never converts an observational study into evidence of causation — that requires random assignment to treatment groups.
What's a confounding variable?
A confounding variable is an outside factor that affects both variables being studied, creating an association between them that isn't a direct cause-and-effect relationship.
Can an experiment ever be biased despite random assignment?
Yes — random assignment protects against confounding variables between treatment groups, but a study can still suffer from a non-random or unrepresentative sample, a small sample size, or poor measurement. Random assignment alone doesn't guarantee a study is flawless; it only guarantees that its causal comparison between groups is fair.
How does this connect to margin of error questions?
Both topics deal with how much confidence to place in a study's result — margin of error addresses precision of an estimate, while study design addresses whether a causal claim is justified at all. See our guide on inference from sample statistics and margin of error for the precision side of this reasoning.
You can practice this exact reasoning style, along with the rest of the domain, through the Problem-Solving and Data Analysis practice hub and free SAT Math practice sets with full worked explanations for every question.