Experimental design, explained with examples
A well-designed experiment is the only way to show that one thing causes another. AP Statistics expects you to recognise good designs, describe how to randomise, and explain what conclusions a study can and cannot support.
On this page
- Experiments versus observational studies
- Key vocabulary
- The four principles of a good experiment
- Completely randomised design
- Randomised block design
- Matched pairs design
- Blinding and the placebo effect
- Worked examples
- Scope of inference
- Common mistakes
- Practice questions
- Frequently asked questions
- Key takeaways
- Related guides
Experiments versus observational studies
In an observational study, researchers observe individuals and measure variables without influencing them, for example recording how much coffee people drink and how well they sleep. In an experiment, researchers deliberately impose treatments on individuals and measure their responses, for example assigning some volunteers to drink coffee in the evening and others not to.
The difference matters because observational studies cannot establish cause and effect. People who drink more coffee might also work longer hours or be more stressed, and those confounding variables could explain the poor sleep. A properly randomised experiment breaks those links.
Key vocabulary
- Experimental units (called subjects when they are people): the individuals that receive treatments.
- Explanatory variable (factor): what the researcher changes.
- Treatments: the specific conditions applied, such as dosages of a drug.
- Response variable: what is measured to see the effect.
- Control group: a group that receives no active treatment or a standard treatment, for comparison.
- Placebo: a fake treatment that looks the same as the real one.
The four principles of a good experiment
- Comparison: use at least two treatment groups, or a treatment and a control, so you can compare.
- Random assignment: assign treatments to units at random. This creates groups that are roughly equivalent in every way, known and unknown, so differences in the response can be attributed to the treatments.
- Control: keep other variables the same for all groups, such as the time of day, instructions and measurement method.
- Replication: use enough experimental units in each group so that real treatment effects can be distinguished from chance variation.
Completely randomised design
All experimental units are assigned to treatments completely at random. A clear AP description: “Number the 60 volunteers 1 to 60. Use a random number generator to select 30 different numbers; those volunteers receive the new diet. The remaining 30 receive the standard diet.” Alternatively, put 60 identical slips, 30 labelled A and 30 labelled B, in a hat, mix thoroughly and let each volunteer draw one.
Randomised block design
If some variable is likely to affect the response, such as age or gender, first group the units into blocks of similar units, then randomly assign treatments within each block. Blocking reduces the variability caused by that variable, making it easier to detect a treatment effect. The rule of thumb: block what you can, randomise what you cannot.
Matched pairs design
A matched pairs design is a special block design with blocks of size two. Either pair up very similar units and randomly assign one treatment to each member of a pair, or give both treatments to each unit in a random order. Comparing the two responses within each pair removes much of the variation between individuals. Matched pairs data is analysed with a paired t procedure on the differences.
Blinding and the placebo effect
People sometimes improve simply because they believe they are being treated: the placebo effect. To guard against this, the control group receives a placebo. In a single-blind experiment, the subjects do not know which treatment they receive. In a double-blind experiment, neither the subjects nor the people measuring the responses know, which also prevents the researchers' expectations from influencing the results.
Worked examples
Example 1: identifying parts
A teacher tests whether listening to music while revising improves test scores. 40 students are randomly assigned to revise with or without music, then all take the same test. Identify the explanatory variable, treatments, response and experimental units.
- Explanatory variable: music while revising. Treatments: music and no music.
- Response: test score. Experimental units: the 40 students.
Example 2: choosing a block design
A company tests two fertilisers on 20 plots, 10 on a sunny hillside and 10 in a shady valley. How should the experiment be designed?
- Sunlight probably affects growth, so block by location: hillside plots in one block, valley plots in the other.
- Within each block, randomly assign 5 plots to each fertiliser, then compare the fertilisers within each block.
Example 3: matched pairs
A shoe company wants to compare the wear of two sole materials. Describe a matched pairs design.
- Give each volunteer one shoe of each material, with the material on the left foot decided by a coin flip.
- Each person acts as their own pair, so differences in how much people walk are removed from the comparison.
Example 4: spotting confounding
A study finds that students who use a revision app score higher. Can we conclude the app causes higher scores?
- No. This is observational. Motivated students may be more likely both to use the app and to revise in other ways: motivation is a possible confounding variable.
- A randomised experiment, assigning the app to some students and not others, would be needed to show cause and effect.
Scope of inference
AP free-response questions often ask what conclusions are justified. Use this two-part check:
- Random assignment to treatments allows conclusions about cause and effect.
- Random selection from a population allows generalisation to that population.
Most experiments use volunteers, so they have random assignment but not random selection. Their results show causation for subjects like those in the study, but should be generalised with care.
Common mistakes
- Confusing random assignment (in experiments) with random sampling (in surveys).
- Claiming causation from an observational study.
- Describing randomisation vaguely; say exactly how random numbers or slips are used.
- Mixing up blocks (formed before randomising, by a variable) with treatment groups.
Practice questions
Try each one before opening the answer.
Q1. Researchers record the diets and health of 5000 adults over ten years. Experiment or observational study?
Show answer
Q2. Why use a placebo in a drug trial?
Show answer
Q3. What does double-blind mean?
Show answer
Q4. When is a block design better than a completely randomised design?
Show answer
Frequently asked questions
Is experimental design on the AP Statistics exam?
Yes. It is part of Unit 3, collecting data, and a design question appears on the free-response section most years.
Is experimental design in A level Maths?
A level Maths focuses on sampling rather than experimental design, but the ideas of control, randomisation and confounding are central to A level science practicals and to university study.
Key takeaways
- Only randomised experiments can show cause and effect.
- Principles: comparison, random assignment, control and replication.
- Block by variables that affect the response; randomise within blocks.
- Matched pairs compare two treatments within similar pairs or the same individual.
- Random assignment gives causation; random selection gives generalisation.