What a study must do before its results can support a claim about cause: the anatomy of a randomized experiment with its explanatory and response variables, treatments and experimental units; lurking variables and why random assignment rather than random selection is what neutralises them; the control group and the placebo that make the power of suggestion measurable; blinding and double-blinding and what each one protects against; and the studies that cannot be randomized at all, where an observational design shows association and cannot establish cause. Closes with the ethical obligations a researcher carries toward the people being studied.
Subject: Statistics · 65 slides · symbolic lesson
Open the interactive version of this deck
Title
Statistics · Chapter 1 — Sampling and Data
What a study has to do before it may say one thing causes another, and what it owes the people it studies
Objectives
Every week a headline says that some food, drug or habit causes something. This lesson gives you the questions that decide whether the study behind the headline could possibly know that, and they turn out to be few and sharp.
The arithmetic in this lesson is light: a few percentages and one subtraction. The weight is in the reasoning, and every conclusion you write should name the feature of the design that entitles you to it. A sentence that says the study proves something, without saying what the design did, is the answer this lesson trains you out of.
Keep one question running under every slide: what else could have produced this difference? A good design is simply a design in which that question has no answer left.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 34-38 — Experimental Design and Ethics
Warm-up
Section 1.1 gave you two words that this lesson leans on in almost every slide: population and sample. Get them straight now, because the first worked example asks for both, and the most common error in this section is mixing them up.
Discussion prompt
A dietitian wants to know whether adults in her city eat enough fibre. She interviews 250 adults at three supermarkets. Name the population and the sample, and say one reason the sample might not look like the population.
Notice the shape of that last answer: it names a specific way the chosen people could differ, and connects it to the thing being measured. That is exactly the move this lesson asks you to make about groups inside a study, not just about the sample as a whole.
Section 1.2 made the same point from the other side: a sample is only as good as the way it was chosen. Here the question moves one step further on. Once the people are chosen, how do you decide who gets which treatment, so that a difference at the end means something?
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.1, p. 6 — population and sample, in the Key Terms
Section
Part 1
Concept
Does aspirin reduce the risk of heart attack? Is one fertilizer better for roses than another? Each question has the same shape: one variable, which you suspect is doing something, and a second variable, which you suspect it is doing it to.
explanatory and response variables — When one variable causes change in another, the first is the explanatory variable and the affected one is the response variable.
\[ \text{explanatory variable} \;\longrightarrow\; \text{response variable} \]
The arrow is the claim under test, not a fact you already have. In the aspirin question the explanatory variable is the medication and the response variable is whether a heart attack happens. In the roses question it is the fertilizer brand and the growth of the roses.
What makes a study an experiment is who controls the left-hand side. In a randomized experiment the researcher manipulates the values of the explanatory variable, deciding who gets what, and then measures the changes in the response. If the researcher only watches what people already do, it is not an experiment, however careful the watching.
A quick test for which variable is which: ask which one could be set in advance. Nobody can hand out heart attacks, but anyone can hand out pills. The one you could hand out is the explanatory variable.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the purpose of an experiment
Concept
Two more words complete the vocabulary. The explanatory variable takes a few different values in a study, and each value has a name of its own.
treatments — The different values of the explanatory variable. If the explanatory variable is fertilizer brand and two brands are compared, there are two treatments.
experimental unit — A single object or individual to be measured: one person, one rose bush, one plot of soil.
The distinction to hold on to is that a treatment is a value, not a group of people. The group that receives aspirin is a treatment group; aspirin is the treatment. When a question asks for the treatments, it wants the list of values, and a list of groups is half an answer.
\[ \text{explanatory variable: medication} \;\Rightarrow\; \text{treatments: aspirin, placebo} \]
Experimental units are usually easy once you ask what one measurement is taken on. In a drug trial each person gives one yes or no at the end, so each person is a unit. In the roses question, one bush gives one growth measurement, so the bush is the unit, and the gardener is not.
When the units are people they are often called subjects or participants. The words are interchangeable in this course.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — treatments and experimental units
Notation
Here is the whole vocabulary of Part 1 in one line. Hover or tap each piece to see which word it stands for and the question that finds it in a described study.
Annotate
On: \( \text{units} \xrightarrow{\text{ treatments }} \text{response} \)
The arrow is the part students skip, and it carries the most weight. It says the researcher hands out the treatments. Part 2 will add a single word above it, randomly, and that word is what turns an observed difference into evidence of cause.
Two further words sit around the outside of this line. The population is who the question is about, and the sample is the units actually used. So a complete description of a study names six things: population, sample, units, explanatory variable, treatments and response variable.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 34-35 — the six parts named in Example 1.19
Picture it
Before the first example, look at the aspirin study as a picture. The order of the steps is part of the design, and one step in particular is the whole reason the conclusion can be trusted.
Figure (svg): A flow diagram of Example 1.19 from left to right: the population of people aged 50 to 84, a sample of 400 recruited, a random-assignment step that splits them, an aspirin group of 200 and a placebo group of 200, each followed by counting heart attacks, and a final comparison.
Read it left to right. The question is about people aged 50 to 84; 400 of them are recruited; a random step splits them; each half takes one pill a day for three years; heart attacks are counted in each group; the two counts are compared.
Notice where the randomness sits: after recruitment, before anything is handed out. A study that recruited people at random and then let them choose their own pill would have randomized the wrong step, and Part 2 shows exactly what that costs.
Notice also that the placebo group is drawn as a treatment group, not as an empty box. Both groups swallow a pill every day for three years. The explanatory variable is the pill, and it takes two values.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Example 1.19
Worked example
Researchers want to know whether taking aspirin regularly reduces the risk of heart attack. They recruit 400 people aged 50 to 84, divide them randomly into two groups, and give one group aspirin and the other a placebo, one pill a day for three years. Nobody knows which pill they take. At the end, the researchers count heart attacks in each group.
Population: who the question is about
Why: the question concerns people in this age band, not only the ones who volunteered.
\[ \text{population: people aged 50 to 84} \]
Sample: who was actually studied
Why: the recruited group, all of them, before the split.
\[ \text{sample: the 400 participants} \]
Experimental units: what one measurement is taken on
Why: each person yields one result at the end, heart attack or not.
\[ \text{units: the individual people} \]
Explanatory variable: what the researcher hands out
Why: the pill each person swallows, which the researchers decided.
\[ \text{explanatory variable: oral medication} \]
Treatments: the values it takes
Why: two values, and the placebo is one of them.
\[ \text{treatments: aspirin, placebo} \]
Response variable: what is recorded at the end
Why: a yes or no for each person.
\[ \text{response: whether the person had a heart attack} \]
The response here is categorical, a yes or no per person, which is why the comparison at the end will be a count or a percentage in each group rather than an average.
Check that the parts fit together
Why: the treatments must be values of the explanatory variable, and the response must be something measured on each unit after treatment. Aspirin and placebo are both values of oral medication, and each person is measured once at the end, so all six answers are consistent.
\[ \text{aspirin, placebo} \in \text{oral medication} \quad \checkmark \]
The commonest slip is to answer aspirin for the explanatory variable. Aspirin is one value; the variable is the medication, and it has a second value, the placebo, that is just as much a part of the design.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Example 1.19
Matching
The same study, the other way round. Each description on the right belongs to exactly one part on the left. Do it without looking back, then say to yourself the question that finds each part.
Match the pairs
Why: The population is who the question is about and the sample is who was studied, so the age band and the 400 go there in that order. One person gives one result, so one person is a unit. The treatments are the two pills handed out, and the response is what is recorded at the end.
If you paired the population with the 400, look again at what the question asks. The researchers want to know about aspirin and heart attacks in everyone of that age, and the 400 are the part of that everyone they could reach.
If you paired the units with the 400, the difference is one of scale. The sample is the collection; a unit is one member of it, the thing a single measurement is taken on. The sample is 400 units.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Example 1.19
Worked example
A study looks at the effect of three medicines, A, B and C, on the height of adults aged 30 to 45. Ninety adults are selected randomly and divided into three equal groups; each group takes one medicine for six months, and the average change in height in each group is calculated at the end.
Population and sample
Why: the question is about adults in this age band; ninety were used.
\[ \text{population: adults aged 30 to 45} \qquad \text{sample: the 90 selected} \]
Group sizes
Why: three equal groups share ninety people.
\[ 90 \div 3 = 30 \text{ adults per medicine} \]
Units
Why: each adult gives one change in height.
\[ \text{units: the individual adults} \]
Explanatory variable and treatments
Why: the book says explanatory variables, but there is one variable here with three values.
\[ \text{explanatory: medicine} \qquad \text{treatments: A, B, C} \]
Response variable
Why: measured on each adult, then averaged within each group.
\[ \text{response: change in height over 6 months} \]
Now look at what is missing. There is no placebo group, so every group receives an active medicine. The design can compare A with B with C, but it cannot say whether any of them does anything at all compared with taking nothing, and adults of 30 to 45 are not expected to grow in any case.
Check the parts against each other
Why: A, B and C are the values of the medicine variable, the response is measured on every unit after six months, and three groups of 30 account for all 90 adults.
\[ 30 + 30 + 30 = 90 \quad \checkmark \]
A tidy answer to this Try It also notes the missing control group. Naming the six parts is the start; asking what the design can and cannot compare is the habit.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Try It 1.19
Sorting
Six variables from studies in this section and its exercises. For each one ask a single question: could the researcher set it in advance and hand it out? If so it is explanatory; if it is recorded at the end, it is the response.
Sort into buckets
Sort each variable by the role it plays in its study.
The pattern across the three studies is worth noticing. The thing a study is about, the headline outcome, is almost always the response. The explanatory variable is usually the plainer-sounding thing the researcher controls, which is why students so often name them the wrong way round.
Units of measurement are a useful clue too. A response is often a time, a count or a yes or no recorded per subject, while an explanatory variable is usually a short list of named options.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 35-36 — Examples 1.19 and 1.20, Try It 1.21
Section
Part 2
Concept
Here is the book's example of a study that looks convincing and proves nothing. You recruit a group of people, ask whether they regularly take vitamin E, and find that the ones who do are healthier on average than the ones who do not.
Figure (svg): A causal diagram: a box on the left reading takes vitamin E, a box on the right reading better health, joined by a dashed arrow with a question mark. Above them three boxes, exercise, diet and does not smoke, each with arrows to both the vitamin E box and the health box.
The trouble is that the two groups differ in far more than vitamin E. People who take a supplement every day are often the same people who exercise, eat carefully, take other vitamins and choose not to smoke. Any one of those could be making them healthier, with the vitamin along for the ride.
Look at what each yellow box in the picture does. Exercise makes someone more likely to take vitamin E and also makes them healthier, so exercise alone would produce the gap the study saw, even if vitamin E did nothing at all. The same is true of diet and of not smoking.
The researcher here chose nothing: the people chose for themselves whether to take the vitamin. That is the root of the problem, and it is why the study cannot be rescued by measuring more carefully.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the vitamin E study
Concept
The yellow boxes have a name. The book defines them in one line, and the definition is worth unpacking because it is stricter than it looks.
lurking variable — An additional variable, not the one being studied, that can cloud a study by offering another explanation for the difference between groups.
To be a rival explanation, a variable has to do two things at once. It has to differ between the groups being compared, and it has to affect the response. A variable that does only one of these clouds nothing.
\[ \text{differs between groups} \;+\; \text{affects the response} \;=\; \text{lurking} \]
Eye colour is a variable, and it certainly differs from person to person, but it does not make anyone healthier, so it cannot explain the vitamin E gap. Exercise passes both tests, which is what makes it dangerous.
To prove that the explanatory variable causes a change, it has to be isolated: the groups must differ in exactly one way, the planned treatment. The rest of this part is about how a design achieves that without knowing what the lurking variables are.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — lurking variables
Worked example
Work the vitamin E observation the way you would work any study you are asked to judge: name its parts, then ask whether anything other than the explanatory variable differs between the groups.
Name the explanatory and response variables
Why: the supplement is suspected of causing better health.
\[ \text{explanatory: takes vitamin E} \qquad \text{response: health} \]
Ask who decided the explanatory variable
Why: the participants did, before the study began.
\[ \text{vitamin E chosen by the participant, not assigned} \]
List what else goes with that choice
Why: habits that travel with taking supplements.
\[ \text{exercise, diet, other vitamins, not smoking} \]
Test each against the two conditions
Why: each differs between takers and non-takers, and each affects health.
\[ \text{differs between groups} + \text{affects health} \Rightarrow \text{lurking} \]
State the conclusion the design allows
Why: an association, with rival explanations still standing.
\[ \text{takers healthier: association, not cause} \]
That conclusion does not say vitamin E is useless. It says this study cannot tell. The vitamin might help, might do nothing, or might even do slight harm that the takers' other habits more than make up for; all three are consistent with what was observed.
Check the conclusion by imagining vitamin E does nothing
Why: if the gap would still appear with a useless vitamin, the study cannot be evidence for it. Exercisers take the vitamin more and are healthier anyway, so a useless vitamin would still show a gap.
\[ \text{useless vitamin} \Rightarrow \text{gap still appears} \Rightarrow \text{no evidence} \]
That imagined check is the most useful test in this lesson. Ask what the data would look like if the treatment did nothing; if the answer is the same as what was seen, the study has shown nothing about the treatment.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the vitamin E study
Concept
The cure is to take the choice away from the subjects. If a coin decides who gets vitamin E, then exercise, diet and smoking have no way to influence who ends up in which group.
Figure (svg): Four dot grids of the same 400 simulated people, with regular exercisers drawn in yellow and sorted to the front. Top row, people who chose vitamin E for themselves: 117 of 182 takers exercise but only 43 of 218 non-takers. Bottom row, people split by a coin: 83 of 200 in one group and 77 of 200 in the other.
These are the same 400 simulated people twice, with the regular exercisers in yellow. In the top row, people chose for themselves, and the exercisers flocked to the vitamin: 117 of the 182 takers exercise, against only 43 of the 218 non-takers. That is 64.3 percent against 19.7 percent.
In the bottom row a coin decided. The exercisers are now split 83 and 77, or 41.5 percent against 38.5 percent. The coin knew nothing about exercise, so exercise could not steer anyone into either group.
This is what the book means by saying that when subjects are assigned treatments randomly, all the potential lurking variables are spread equally among the groups. It works for exercise, and it works in exactly the same way for every variable nobody thought to measure, because the coin ignores all of them at once.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — random assignment spreads lurking variables among the groups
Worked example
The two coin groups were not identical: one had 83 exercisers and the other 77. Work out how big that leftover gap is, and compare it with the gap self-selection produced, using the counts from the dot grids.
Self-selected: exerciser share among the takers
Why: 117 of the 182 who chose vitamin E.
\[ \dfrac{117}{182} = 0.643 = 64.3\% \]
Self-selected: share among the non-takers
Why: 43 of the 218 who did not.
\[ \dfrac{43}{218} = 0.197 = 19.7\% \]
The self-selected gap
Why: subtract the shares, unrounded, and round at the end.
\[ 64.29 - 19.72 = 44.6 \text{ percentage points} \]
Coin-assigned: the two shares
Why: 83 and 77 exercisers, 200 people in each group.
\[ \dfrac{83}{200} = 41.5\% \qquad \dfrac{77}{200} = 38.5\% \]
The coin-assigned gap
Why: a simple subtraction this time, since both groups have 200 people.
\[ 41.5 - 38.5 = 3.0 \text{ percentage points} \]
So the coin left a gap of 3 points and self-selection left one of 44.6. Random assignment does not make the groups identical; it makes the leftover difference small and, just as important, not tilted towards either treatment.
Check the counts against the whole
Why: the exercisers in each pair of groups must add back to the 160 exercisers among the 400.
\[ 117 + 43 = 160 \qquad 83 + 77 = 160 \quad \checkmark \]
Both pairs account for all 160 exercisers, so the two splits really are of the same 400 people, and the only thing that changed between them is who made the choice.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — random assignment and lurking variables
Picture it
One coin split left a gap of 3 points. Was that luck? Re-run the coin a thousand times on the same 400 people and record the gap in exerciser share each time.
Figure (svg): A histogram of the gap in exerciser share between two coin-assigned groups over 1,000 re-draws: a mound centred on zero running from about minus 18 to plus 16 points. Far to the right, a dashed line at 44.6 points marks the gap produced by self-selection.
The re-draws pile up around zero. In this run, 734 of the 1,000 left a gap of 5 points or less and 963 left a gap of 10 or less; the most lopsided split of all was 18 points. That is how big chance imbalances get with 200 people a group.
Now look at the dashed red line at 44.6, where self-selection put the gap. Not one of the thousand coin splits gets within 25 points of it. Self-selection does not produce an unlucky split; it produces a split of a completely different kind.
This picture is the seed of most of the rest of the course. Later chapters measure exactly how wide this mound is, and use it to decide whether a difference between treatment groups is bigger than the coin alone would produce.
Tweak it
The mound on the last slide had a typical width of about 5 points for groups of 200 people. The formula in the readout gives that typical width for any group size and any share of people with the lurking trait; you do not need to derive it, only to read it.
Parameter explorer
Drag the share p of people with the trait, then read the curve: how large is the typical chance gap for groups of 200, and for groups of 20?
\[ \text{typical gap} = 100\sqrt{\dfrac{2 \cdot {p}(1 - {p})}{n}} \text{ points} \]
At p of 0.4 and groups of 200 the formula gives 4.9 points, which matches the thousand re-draws, whose spread came out at 4.95. The formula and the simulation are two views of the same coin.
Now slide along the curve. With groups of 20 the typical gap is 15.5 points, big enough to matter; with groups of 1,000 it is 2.2. Random assignment balances lurking variables better the more units it has to work with, which is one honest reason experiments want many subjects.
Keep this separate from what comes next. Here, a bigger experiment shrinks a gap that was already centred on zero. On the next slide, a bigger observational study shrinks nothing that matters, because its gap was never centred on zero.
Concept
A tempting reply to the vitamin E problem is to survey more people. Surely forty thousand people outweigh a few lurking variables? The simulation below says no, and says why.
Figure (svg): Dots for simulated studies of sizes 100 to 40,000 on a log scale. Orange dots, observational studies of people who chose vitamin E, scatter widely at first and settle on a dashed line at 11.6 points. Blue dots, randomized experiments, settle on the zero line. In the simulation vitamin E has no effect at all.
In this simulated world vitamin E does nothing at all; health depends only on exercise. Each orange dot is an observational study of people who chose for themselves, each blue dot a randomized experiment, at sizes from 100 to 40,000.
Small studies of either kind scatter widely. As they grow, both kinds steady down, but to different places. The randomized studies settle on zero, the truth. The observational studies settle on 11.6 points, the gap exercise creates: at 40,000 people the orange dot still sits at 12.7 and the blue one at 0.3.
So size buys precision, not honesty. A large observational study measures the association very precisely, and the association includes every lurking variable. Only the coin removes them, which is why the book says an experiment isolates the explanatory variable by random assignment and not by numbers.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — isolating the explanatory variable
Prediction
Commit to an answer before reasoning it through; the last slide gave you everything you need, but it is worth finding out whether the idea has landed.
Predict first
An observational survey of 5,000 people finds vitamin E takers 12 points healthier. The researchers repeat it with 10,000 people chosen the same way. What should you expect?
Correct: A gap near 12 points again, measured a little more precisely
Why: Doubling the sample does nothing to who chose the vitamin, so the exercisers still crowd into the taker group and the lurking gap is still there. A bigger sample only pins that same gap down more precisely.
If you chose the answer that says the gap shrinks towards zero, you have mixed up the two effects of size. A bigger sample shrinks random scatter, the wobble between one study and the next. It does not shrink a gap that is built into the way the groups were formed.
The answer that says the gap doubles confuses a count with a percentage. The gap is a difference in percentages, and percentages do not grow because the sample does.
The last option is the one to reject most firmly. What makes a study an experiment is who assigns the treatment, never how many people take part. A survey of a million volunteers who chose their own vitamins is still an observational study.
Discrimination
Two different random steps can appear in a study, and they do different jobs. For each description, ask what the randomness decided: who is in the study, or what they receive once they are in it.
Sort into buckets
Sort each description by the job its randomness does.
A study can have either kind of randomness, both, or neither. The aspirin study used random assignment but recruited volunteers, so its conclusion about cause is strong while its reach to all people aged 50 to 84 depends on how typical the volunteers were.
Keep the two jobs in one sentence each. Random selection earns generalisation: the sample looks like the population. Random assignment earns causation: the groups look like each other.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.2, p. 18 — random sampling
Trap
Here is a conclusion that sounds careful and is wrong. The survey really did use a random sample, which is exactly why the error is so persuasive.
\[ \text{random sample of adults} \]
\[ \text{takers are healthier} \]
\[ \Rightarrow \text{ vitamin E works} \]
The randomness in this study decided who was surveyed. It did not decide who takes vitamin E; the adults decided that themselves, long before anybody called.
Name the randomness correctly, and the conclusion shrinks to what the design supports.
\[ \text{random selection:} \]
\[ \text{sample represents adults} \]
\[ \text{self-chosen vitamin E:} \]
\[ \text{association only} \]
A random sample is valuable: it means the association is probably true of adults in general, not just of an odd group. But the association is still an association, with exercise, diet and smoking tangled up in it.
To earn the word causes, the researcher must be the one who decides who takes vitamin E, and must decide it at random. When you read a study, find the random step and ask what it decided.
Counterexample
A study reports that people who eat breakfast weigh less, on average, than people who skip it, and concludes that eating breakfast prevents weight gain. The observation may be perfectly accurate. Your job is to break the conclusion.
Discussion prompt
Name two lurking variables that could produce this gap even if breakfast did nothing, showing for each that it differs between the groups and affects weight. Then describe the study that would settle the question.
Notice that each lurking variable needed both halves of the test. A variable that only affects weight, or only affects who eats breakfast, cannot produce the gap on its own.
Notice too that you did not need to know which lurking variable was really at work. That is the power of the coin: the settling study protects against the rivals you named and the ones you never thought of, all at once.
Section
Part 3
Concept
Random assignment handles the differences people bring with them. There is a second problem it does not handle: what happens to people because they know they are in a study. The book calls it the power of suggestion, and quotes a study of a performance-enhancing substance.
Figure (svg): A two by two grid. Columns: given the real substance, drawn as a filled pill, and given an inert pill, drawn as a hollow pill. Rows: told they took the substance, and told they did not. The two cells where subjects believed they had taken it are shaded bright and marked fast; the cell given the drug without knowing is marked no significant gain.
The study crossed two things: what athletes actually received, and what they were told. Those who believed they had taken the substance ran almost as fast as those who really had it. Those who got the substance without knowing showed no significant gain.
Read the grid along the top row and then down the left column. Along the top, belief does nearly all the work. Down the left, the substance without belief does almost nothing. In a study where everyone who got the drug also knew it, the two would be completely tangled.
So a treatment group that knows it is treated is measuring two things at once: the treatment, and the effect of expecting the treatment to work. Part 3 is about separating them.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the power of suggestion, McClung and Collins 2007
Concept
To measure the power of suggestion, researchers set one group aside as a control group and give it a placebo. The book's definition of a placebo is precise, and one word in it surprises people.
placebo — An active treatment that cannot directly influence the response variable: a sugar pill, a saline injection, a sham procedure.
Active, because the control group really does something: swallows a pill every day, visits the clinic, fills in the forms. Everything about being in the study happens to them. Only the ingredient that could act on the response is missing.
\[ \text{treatment group: study + ingredient} \qquad \text{control: study only} \]
That is exactly why the control group is useful. It experiences everything the treatment group does except the one thing under test, so the difference between the groups is the effect of that one thing. The book puts it as balancing the effects of being in an experiment against the effects of the active treatment.
A control group that got nothing at all, no pill and no visits, would not do this job. It would differ from the treatment group in two ways, the ingredient and the whole experience of being treated, and a difference at the end could come from either.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — the control group and the placebo
Notation
Here is the subtraction every placebo-controlled study ends with. Tap each piece to see what it contains; the point is that one ingredient appears on both sides and cancels.
Annotate
On: \( \underbrace{55\%}_{\text{treated}} - \underbrace{42\%}_{\text{placebo}} = 13 \text{ points} \)
Written out, the argument is one line of algebra. If both arms share the same study effect, subtracting one arm from the other removes it.
\[ (\text{drug} + \text{study}) - (\text{study}) = \text{drug} \]
That cancellation only happens if the study effect really is the same in both arms. It is the same only if both arms believe the same thing about what they are taking, which is where blinding, in Part 4, comes in.
Worked example
A sleep supplement is tested against a placebo. Participants are randomly assigned, and neither they nor the staff know who has which pill. After four weeks, 55 percent of the supplement arm and 42 percent of the placebo arm report sleeping better. Report the supplement's effect.
Figure (svg): Two bars: a placebo arm at 42 percent reporting better sleep and a supplement arm at 55 percent. A dashed line at 42 runs across both bars; the part of the supplement bar above it, 13 points, is highlighted as the supplement's own effect.
Name what the placebo arm shows
Why: 42 percent improved on a pill that cannot act on sleep: the effect of being in the study.
\[ \text{placebo arm: } 42\% \]
Subtract to isolate the supplement
Why: the study effect is common to both arms, so it cancels.
\[ 55\% - 42\% = 13 \text{ percentage points} \]
Say it as a relative change too
Why: 13 points on top of a base of 42.
\[ \dfrac{13}{42} = 0.3095 \approx 31\% \text{ more people improving} \]
Both numbers are correct, and they answer different questions. Thirteen points is the absolute effect: out of every hundred people, thirteen more improved. Thirty-one percent is the relative effect, and it sounds larger, which is why advertisements prefer it.
Write the conclusion in context
Why: the random assignment and the placebo together license a causal sentence.
\[ \text{the supplement raised the share sleeping better by 13 points} \]
What you must not write is that the supplement helped 55 percent of people. Forty-two of those 55 points would have happened with a sugar pill.
Check by adding the pieces back
Why: the placebo arm's rate plus the supplement's effect should rebuild the treated arm's rate.
\[ 42 + 13 = 55 \quad \checkmark \]
The subtraction and the addition agree, so the 13 points is exactly the part of the treated arm the placebo arm cannot account for.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — the control group balances the effect of being in an experiment
Estimation
Suppose the same trial had been run with no control group at all: everyone took the supplement, and 55 percent reported sleeping better. Estimate what that trial would have announced, compared with the truth from the last slide.
Predict first
With no placebo arm, how would the reported effect compare with the supplement's real effect of 13 points?
Correct: It would overstate the effect several times over
Why: With nobody to compare against, all 55 percent of the improvement looks like the supplement's doing, while the placebo arm showed that 42 of those points happen with a sugar pill. The real effect is 13 points, so the uncontrolled trial credits the supplement with more than four times its share.
The ratio is worth computing: 55 divided by 13 is about 4.2. An uncontrolled trial of this supplement would have credited it with more than four times the effect it really has, and it would have looked like strong evidence.
This is why the words placebo-controlled matter on a drug label. An improvement rate on its own tells you what happened to the treated people, not what the treatment did to them.
The other options each miss the point in a different way. The uncontrolled trial does say something, just the wrong thing. And it cannot understate, because it adds the study effect to the drug effect instead of removing it.
Trap
Asked for the treatments in the aspirin study, many students write one treatment and one empty group.
\[ \text{treatment: aspirin} \]
\[ \text{control group: untreated} \]
It sounds natural, because the control group receives no aspirin. But it misdescribes what the control group experiences, and it misleads you about what the comparison measures.
The control group takes a pill every day for three years, exactly as the aspirin group does. That pill is a treatment.
\[ \text{explanatory: oral medication} \]
\[ \text{treatments: aspirin, placebo} \]
Getting this right matters because the whole value of the control group lies in how much it shares with the treatment group. Describe it as untreated and you invite the design error that comes with the word: a control group given nothing at all, which cannot separate the ingredient from the experience of being treated.
The book's wording is the one to keep: the control group is given a placebo treatment, an active treatment that cannot directly influence the response variable.
Worked example
The Placebo Research Group wants to find the extent of placebo effects. Randomly selected men take a test, take a pill that induces a mild headache, and take the test again. For half of the men, chosen at random, the pill is replaced by a similar pill with no effect. The researchers record the change in test time.
Explanatory variable
Why: what the researchers decide for each man.
\[ \text{explanatory: which pill is taken} \]
Response variable
Why: measured on each man, before and after.
\[ \text{response: change in time to complete the test} \]
Treatments
Why: the two values of the pill variable.
\[ \text{treatments: headache pill, look-alike inert pill} \]
Lurking variables
Why: the pill was assigned at random, so habits, health and test skill are spread across both halves.
\[ \text{random assignment} \Rightarrow \text{no systematic lurking variable} \]
One thing does still affect both groups: every man takes the test twice, and people usually do better the second time. That practice effect is real, but it is the same in both halves, so it cancels when the halves are compared, exactly as the study effect did on the last worked example.
Blinding
Why: the pills look alike, so the men need not know which they had, and neither need the researchers timing them.
\[ \text{subjects and timers can both be blinded} \]
There is a catch worth raising. A man who develops a headache may guess which pill he had, and that guess could leak back into his performance. A good design would ask afterwards which pill each man thought he got, to see whether the blinding held.
Check the design against its purpose
Why: to measure the placebo effect, the comparison must differ only in the pill's active ingredient. Random assignment and look-alike pills make that the only planned difference.
\[ \text{only planned difference: the active ingredient} \quad \checkmark \]
Notice how the pieces line up with the goal. The design is aimed at the placebo effect itself, so the inert pill here is not a background control; it is half of the thing being studied.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Try It 1.20
Section
Part 4
Concept
A placebo has a weakness built into it. If you are in a study and you know your pill contains no medicine, the power of suggestion is no longer at work on you, and the control group stops measuring what it was meant to measure.
blinding (masking) — A person involved in a study is blinded when they do not know who is receiving the active treatment and who the placebo.
Blinding the subjects is what makes the cancellation in Part 3 legitimate. Both arms believe the same thing, that they might be getting the real treatment, so both arms carry the same expectation, and subtracting one from the other removes it.
\[ \text{blinded subjects} \Rightarrow \text{same expectation in both arms} \]
The book puts this neatly: blinding preserves the power of suggestion. It does not remove it; it makes sure it acts equally on everybody, so that it can be subtracted away.
A study in which only the subjects are blinded is called single-blind. That protects against the subjects' expectations, but a study has other people in it who might also expect something.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — blinding or masking
Concept
The researchers who work with the subjects have expectations too. A researcher who knows which patients are on the new drug may, without meaning to, encourage them more, record a borderline symptom more kindly, or time a maze with a slightly quicker thumb.
Figure (svg): A grid of eyes. Three rows: open study, single-blind, double-blind. Two columns: the subject and the researcher who works with the subject. In the open study both eyes are open; in single-blind the subject's eye is shut and the researcher's open; in double-blind both are shut.
Read the grid by rows. In an open study everyone can see who got what. In a single-blind study the subject cannot. In a double-blind study neither the subjects nor the researchers working with them can, and the book defines it exactly that way.
double-blind experiment — An experiment in which both the subjects and the researchers involved with the subjects are blinded.
Each shut eye closes one route by which expectation could leak into the response. Blinding the subject protects against the power of suggestion. Blinding the researcher protects against the researcher's hopes shaping what gets measured or how.
Someone has to hold the key, of course, or the results could never be sorted. The allocation list is kept by a person who has no contact with the subjects, and it is opened only when the measurements are complete.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — double-blind experiments
Worked example
A foundation studies whether smell affects learning. Each subject completes pencil and paper mazes six times while wearing a mask: three times with a floral-scented mask and three times with an unscented one. Whether the floral mask comes in the first three trials or the last three is decided at random. Researchers record each maze time and the subject's impression of the scent.
Figure (svg): A design diagram for the mask study. All subjects enter a random-assignment box, which splits them into two sequences of six maze trials. Sequence one wears the floral mask for trials 1 to 3 and the unscented mask for 4 to 6; sequence two does the reverse.
(a) Explanatory and response variables
Why: the scent is set by the researchers; the maze time is recorded.
\[ \text{explanatory: scent} \qquad \text{response: time to finish the maze} \]
(b) Treatments
Why: the two values of scent.
\[ \text{treatments: floral mask, unscented mask} \]
(c) Lurking variables
Why: every subject receives both treatments, so the groups cannot differ in who is in them.
\[ \text{same people under both masks} \]
That is a different design from the aspirin study. Instead of comparing two groups of people, each person is compared with themself, so differences in skill, age or eyesight cannot favour either mask. What is randomized is the order, and the next slide shows why the order needed randomizing.
(d) Can the study be blinded?
Why: the subjects can smell the flowers, so they cannot be blinded; the researcher timing the maze need not know which mask is on.
\[ \text{subjects: no} \qquad \text{timing researcher: yes} \]
Check that each answer matches the design
Why: scent is what the researchers change between trials, maze time is measured on every trial, and the random order is the only random step, so it is the one that protects against lurking variables.
\[ \text{randomized: the order} \Rightarrow \text{practice cannot favour a mask} \quad \checkmark \]
The subjects' impression of the scent was recorded as well, but it is a second response, not a treatment: nobody assigned it.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 35-36 — Example 1.20
Picture it
Suppose the designers had not randomized, and every subject wore the floral mask for the last three trials. The curve below is an illustration of a practice effect, with the scent doing nothing at all.
Figure (svg): An illustrative practice curve: maze time falling from 70 seconds on trial 1 to about 42 on trial 6, with dots at each trial. Trials 4 to 6 are shaded pink as if the floral mask were always worn last; their average, 44.4 seconds, sits well below the 59.7-second average of trials 1 to 3.
People get faster at mazes with practice. On this curve the first three trials average 59.7 seconds and the last three average 44.4 seconds, a gap of about 15 seconds that comes entirely from practice.
\[ 59.74 - 44.41 = 15.3 \text{ seconds} \]
If the floral mask always came last, all 15 of those seconds would be credited to the scent. Practice would be a lurking variable in the purest form: it differs between the floral and unscented trials, and it affects maze time.
Randomizing the order breaks that link. Half the subjects meet the scent early and half meet it late, so the practice advantage lands equally on both masks and cancels in the comparison. The book's answer, that random assignment eliminates the problem of lurking variables, is this picture in one sentence.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Example 1.20, part c
Comparison
Four devices have appeared so far, plus the random sampling of Section 1.2. Each exists to answer one specific threat to a study's conclusion. Fill in the threat for each one; one word or a short phrase is enough.
Comparison matrix
| design device | what it does | the threat it answers |
|---|---|---|
| random assignment | a coin decides who gets which treatment | lurking variables |
| placebo for the control group | both groups have the full study experience | power of suggestion |
| blinding the subjects | subjects do not know which they got | subjects' expectations differing between groups |
| blinding the researchers | staff do not know who got what | researcher bias |
| random selection | chance decides who is in the sample | unrepresentative sample |
Read down the last column and you have the checklist for judging any study. When someone tells you a result, go through it: were there lurking variables, could suggestion explain it, could the staff have nudged it, and does the sample represent anyone beyond itself?
Notice which row is different in kind. The first four protect the comparison inside the study, which is what the claim of cause rests on. Random selection protects the reach of the result outside the study, which is what generalisation rests on.
A study can pass the first four and fail the last, like a well-run trial on volunteers from one hospital. It then shows cause convincingly, for people like its volunteers.
Worked example
You want to know how texting affects driving. Design a study of how many seconds a driver takes to respond when the car ahead brakes, while texting and while driving only.
(a) Explanatory and response variables
Why: texting is what you control; the reaction time is what you record.
\[ \text{explanatory: texting or not} \qquad \text{response: reaction time (s)} \]
(b) Treatments
Why: the two values of the explanatory variable.
\[ \text{treatments: text and drive, drive only} \]
(c) Choosing participants
Why: licensed drivers who already text, over a range of ages, tested in a driving simulator so nobody is put in danger.
\[ \text{licensed drivers who text; simulator only} \]
Figure (svg): Two design panels. Left, two separate groups: eight drivers who only drive, eight others who text, each drawn as a person. Right, the same eight drivers each tested twice, once texting and once not, with each person's two sessions joined and a coin deciding which comes first.
(d) Two random groups: good or not?
Why: it works, but people's reaction speeds differ so much that the difference between people can swamp the difference texting makes; testing each driver under both conditions removes it.
\[ \text{better: each driver does both, order random} \]
(e) Lurking variables
Why: whatever differs between the conditions and affects reaction time.
\[ \text{experience, age, fatigue, practice, simulator familiarity} \]
With every driver doing both, experience and age are identical across the two conditions, because they are the same people. Practice and fatigue remain, because the second session comes after the first, and randomizing which session comes first spreads them evenly, exactly as in the mask study.
(f) Blinding
Why: drivers know when they are texting, so they cannot be blinded; the person scoring the reaction times can be, if the recordings are unlabelled.
\[ \text{drivers: no} \qquad \text{scorer: yes} \]
Check the design against the lurking list
Why: each variable named in (e) must be handled by something in the design. Same drivers in both conditions handles experience and age; the random order handles practice and fatigue; the simulator is the same for everyone.
\[ \text{every lurking variable matched to a device} \quad \checkmark \]
That is the discipline of design in one move: list the rivals, then point to the feature of the study that disarms each one. Any rival with nothing pointing at it is a hole.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Try It 1.21
Error analysis
A clinic plans to test a new pain cream. Read the plan, then tap each part of the line below to see where it fails and which device from the last few slides would fix it.
\[ \text{Patients choose cream or no cream; the doctor rates pain} \]
Annotate
On: \( \text{choose} \;\to\; \text{no cream} \;\to\; \text{doctor rates} \)
Every part of the plan fails, and each failure is a different one of the threats you have met. That is common: a design put together without the checklist usually trips on all of them together.
The repaired plan uses the same words the book uses for the aspirin study: patients randomly assigned to the new cream or an identical placebo cream, with neither patients nor the rating doctor knowing which. Randomized, placebo-controlled, double-blind; each adjective answers one of the three failures.
Pain is especially exposed to all three failures because it is reported by the patient and judged by the doctor. When a response is a judgement rather than a measurement, blinding matters most.
Step zero
Before choosing a design, it pays to ask which blinding is even possible, because some treatments announce themselves. Decide this first, and the rest of the design follows from it.
Discussion prompt
For each proposed study, say whether the subjects can be blinded and whether the people measuring the response can be: (1) a new blood-pressure tablet against a look-alike placebo; (2) a yoga class against no class, with flexibility measured; (3) surgery against a sham incision, with recovery rated by a nurse.
The pattern is the same as in the mask study. When the subjects cannot be blinded, you can nearly always still blind the person who measures, and that is worth doing because it removes one of the two routes by which expectation leaks in.
The surgery case also shows that blinding is a choice about roles. The person who must know, the surgeon, is kept away from the measurement, and the person who measures is kept away from the knowledge.
Section
Part 5
Concept
Everything so far has assumed the researcher can decide who gets which treatment. For many of the questions people care most about, that is impossible, or it would be wrong to try.
Nobody can assign a person's birth order, sex or childhood. Nobody may ethically assign people to smoke for twenty years to see what happens to their lungs. In these cases the researcher can only observe the values people already have, and the study is an observational study.
\[ \text{cannot assign} \Rightarrow \text{observational study} \Rightarrow \text{association} \]
An observational study is not worthless. It can show that two variables go together, that smokers have more lung cancer, that firstborns score differently on some test, and it can measure how strongly. What it cannot do on its own is rule out the rival explanations, because nothing was done to spread the lurking variables across the groups.
When a question can only be studied by observation, the honest word for the result is association. Stronger claims need the kind of evidence that piles up from many observational studies pointing the same way, with the plausible lurking variables measured and accounted for, as happened with smoking.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Example 1.21
Worked example
A researcher wants to study the effect of birth order on personality. Explain why this study could not be conducted as a randomized experiment, and name the main problem with a study that cannot be.
Figure (svg): A family timeline. Three children are born when the parents are 25, 29 and 34, marked firstborn, middle and last-born. Under each child, what else differs: parents' age, number of older siblings, and the household's circumstances at that time. A red crossed-out coin shows that birth order cannot be assigned.
Identify the explanatory variable
Why: birth order is what is suspected of shaping personality.
\[ \text{explanatory: birth order} \qquad \text{response: personality} \]
Ask whether it can be assigned
Why: no coin can decide who is born first; people arrive with their birth order.
\[ \text{birth order cannot be randomly assigned} \]
Name what is lost
Why: random assignment is what spreads lurking variables evenly, and without it they stay put.
\[ \text{no random assignment} \Rightarrow \text{lurking variables remain} \]
Name some of them
Why: the timeline shows what travels with birth order.
\[ \text{parents' age, number of siblings, family income at the time} \]
Each of those passes both halves of the lurking test. A firstborn has younger, less experienced parents and no older siblings; a last-born has neither of those. Any of these differences could shape personality, and all of them come packaged with birth order.
Check the answer against the book's principle
Why: when subjects cannot be assigned to groups at random, the groups differ in more than the explanatory variable, which is exactly the problem named above.
\[ \text{groups differ in more than birth order} \quad \checkmark \]
So the most such a study can report is that personality differs with birth order, an association, and a careful report lists the lurking variables it could not rule out.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Example 1.21
Discrimination
Six research questions. For each, decide whether a researcher could randomly assign the explanatory variable, both practically and ethically. If not, the question can only be studied by observation.
Sort into buckets
Sort each question by the kind of study that can answer it.
Two different reasons land questions in the second group. For sibling status and home address the obstacle is practical: the variable cannot be assigned. For drinking in pregnancy the obstacle is ethical: it could be assigned, but assigning a likely harm is forbidden.
The ethical case links to Part 6. The rules that protect subjects are one of the main reasons that so much of what we know about health comes from observational studies, and why those studies have to be read so carefully.
Trap
A news story reports an observational study of 20,000 adults with the headline below. The numbers in it are genuine; the conclusion is the problem.
\[ \text{coffee drinkers live longer} \]
\[ \Rightarrow \text{ drink coffee to live longer} \]
The step from the first line to the second assumes that coffee is the only difference between drinkers and non-drinkers.
Nobody assigned the coffee. Ask what else differs between people who choose to drink it and people who do not.
\[ \text{drinkers differ in} \]
\[ \text{income, health, habits} \]
\[ \text{observational study:} \]
\[ \text{association only} \]
People who are already ill often give up coffee, which alone would make the remaining drinkers look healthier. Income and work patterns travel with coffee too. Any of these could produce the gap.
The honest sentence is that in this study coffee drinkers lived longer on average, and the study cannot tell whether coffee is the reason. Twenty thousand people make that association precise; they do not make it causal.
Real world
This is the skill the lesson exists for, used on the world rather than the textbook. Find a real health or lifestyle claim in the news this week and look up how the study behind it was done.
Discussion prompt
Say whether the study assigned the explanatory variable. If it did, say whether it used a placebo and blinding; if it did not, name one plausible lurking variable. Then write the one sentence you would use to report the finding honestly.
Look for the telltale words as you read. Linked to, associated with and tied to usually signal an observational study; phrases such as randomly assigned or placebo-controlled trial signal an experiment.
Notice too how often the headline is stronger than the study. The research article may say associated with while the headline says causes. When the two disagree, trust the design, not the headline.
Section
Part 6
Concept
The second half of the section turns from how a study should be designed to how it should be conducted. The book's central case is the social psychologist Diederik Stapel, a professor at Tilburg University, whose fraud led to retractions from some of the world's leading journals.
An investigation across three universities found falsified data in over 55 of his papers and in 10 doctoral dissertations he had supervised. The committee listed what he had done:
Stapel said he was frustrated by messy data that rarely led to clear conclusions, and wanted results that looked elegant. That is worth pausing on, because the pull he describes is one every researcher feels: real data is untidy, and tidy results are rewarded.
The investigation also noted that statistical flaws in his work revealed a lack of familiarity with elementary statistics, and that many co-authors should have spotted the irregularities but trusted him instead. Knowing the basics is a defence, not just a qualification.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 36-37 — the Stapel investigation
Concept
Some misconduct is much harder to spot than invented data. The book mentions researchers who simply stop collecting data once they have just enough to show what they hoped. Every number they report is real; the stopping rule is what is dishonest.
Figure (svg): The running share of improved subjects for one simulated researcher testing a treatment that does nothing, from subject 1 to subject 100. The share starts high, is exactly 60 percent at subject 20, where a dot marks the point the researcher could stop, and then drifts down to 50 percent at subject 100.
Here is one simulated researcher testing a treatment that does nothing: each subject improves with the same chance as a coin landing heads. The researcher checks after every ten subjects and will stop the moment the share improved reaches 60 percent.
After 20 subjects, 12 have improved, exactly 60 percent, and the researcher stops and reports a success. Had the study run to its full 100 subjects, the share would have come back to 50 of 100, exactly what a useless treatment gives.
Early shares jump about because each new subject moves a small total a long way. A researcher who is allowed to choose the moment of stopping is choosing among those jumps, and will naturally choose a flattering one. The honest remedy is to fix the number of subjects before the study begins.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — stopping data collection once the result is favourable
Worked example
One lucky path proves little. Simulate 10,000 researchers, each testing a treatment that does nothing, and count how many can claim success under two rules. Success means at least 60 percent of subjects improved.
Rule 1: fix the sample at 100, look once
Why: count the simulated researchers reaching 60 of 100.
\[ \dfrac{299}{10\,000} = 0.030 = 3.0\% \]
Rule 2: look after every 10 subjects, stop on success
Why: count those who reach 60 percent at any look.
\[ \dfrac{4\,817}{10\,000} = 0.482 = 48.2\% \]
Compare the two rates
Why: how many times more often does peeking produce a false success?
\[ \dfrac{4\,817}{299} = 16.1 \]
With the treatment doing nothing at all, the fixed rule is fooled about 3 times in a hundred. The peeking rule is fooled nearly half the time, about sixteen times as often, and every one of those false successes would be reported with real data.
Verify the simulation with an exact calculation
Why: the chance of 60 or more heads in 100 fair tosses, and the chance of hitting 60 percent at any of the ten looks, can be computed exactly instead of simulated.
\[ \text{exact: } 0.0284 \text{ and } 0.4805 \quad \text{simulated: } 0.030 \text{ and } 0.482 \]
The exact values sit within a fraction of a point of the simulated ones, so the sixteen-fold inflation is a property of the peeking rule, not of an unlucky simulation.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — stopping once there is just enough
Concept
When a study uses human participants, ethics and the law both require researchers to look after them. In the United States the Department of Health and Human Services oversees the federal regulations, and every university or research institution must set up an oversight committee called an Institutional Review Board, or IRB.
Figure (svg): A flow diagram: study plan, then IRB review weighing risks against benefits, then written informed consent from each participant, then data collection, then data guarded for privacy. A dashed arrow runs back from data collection to the IRB, labelled unexpected risks are reported.
Every planned study must be approved in advance by the IRB. The law mandates three key protections, and the picture shows where each one bites:
These sound simple and turn out to be hard. Is deleting names enough to protect privacy, when age, postcode and job together might identify someone? What happens when a study does not go as planned and a risk appears that nobody foresaw? The dashed arrow in the picture is the answer to the second question: the study goes back to the board.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — Institutional Review Boards and the three protections
Worked example
A researcher is collecting data in a community. For each of three things they do, describe the unethical behaviour, how it could affect the reliability of the data, and how it should be corrected.
Figure (svg): A street map of four blocks of houses. One block, where the researcher knows the residents, is outlined in yellow as the only block visited. On it, four houses where no one answered are drawn hollow, and four houses skipped and then filled in with copied answers are marked with red crosses.
(a) Chooses a block where they know many residents
Why: a convenience sample, chosen for comfort, and then presented as the community.
\[ \text{convenience sample} \Rightarrow \text{possibly biased} \]
The people on a familiar street are likely to resemble the researcher and each other, in income, age or background. Claiming they represent the community is misleading. The fix is to choose the areas to visit at random.
(b) Nobody home at four houses; not recorded, not revisited
Why: intentionally leaving out part of the sample.
\[ \text{omitted houses} \Rightarrow \text{missing a whole kind of household} \]
Houses empty in the afternoon are often homes where the adults work. If the survey is about jobs and child care, those are exactly the households it most needs. The fix is to record the addresses and return until the residents are reached.
(c) Skips four houses and fills in answers copied from neighbours
Why: fabricated data, even though each answer was once given by someone real.
\[ \text{duplicated answers} \Rightarrow \text{fraud, and more bias} \]
Check each case against the three failures
Why: (a) biased the choice of sample, (b) biased who was left in it, and (c) invented data; the fixes are random selection, follow-up, and actually interviewing everyone on the route.
\[ \text{(a) selection} \quad \text{(b) omission} \quad \text{(c) fabrication} \quad \checkmark \]
Notice how the three shade into one another. Only one of them is outright fraud, yet all three make the data misrepresent the community, and all three could look perfectly clean in the final spreadsheet.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 38 — Example 1.22
Worked example
A study is commissioned to find the favourite brand of fruit juice among teens in California. For each feature below, describe any unethical behaviour, how it could affect the data, and how to correct it.
(a) Commissioned by the seller of a popular apple juice
Why: a conflict of interest: the sponsor wants a particular answer.
\[ \text{sponsor's interest} \Rightarrow \text{disclose it; keep design independent} \]
(b) Only apple and cranberry juice are included
Why: a question about a favourite brand that offers two juice types cannot find the favourite.
\[ \text{2 options} \Rightarrow \text{answer forced; include the real range} \]
(c) Participants see the brand as samples are poured
Why: brand loyalty and advertising can decide the taste test.
\[ \text{brand visible} \Rightarrow \text{blind the tasting} \]
(d) The results: add the categories the advert combines
Why: 25 percent prefer Brand X and 42 percent have no preference.
\[ 25\% + 42\% = 67\% \text{ like X at least as much} \]
Figure (svg): Left: three bars, prefer Brand X 25 percent, prefer Brand Y 33 percent, no preference 42 percent. Right: two stacked bars. Brand X as much as or more than Y stacks 25 and 42 to make 67; Brand Y as much as or more than X stacks 33 and 42 to make 75.
Combine the same way for Brand Y
Why: 33 percent prefer Y, plus the same 42 percent with no preference.
\[ 33\% + 42\% = 75\% \text{ like Y at least as much} \]
So the commercial's sentence is technically true, and the identical argument gives Brand Y a bigger number. Only 25 percent actually prefer Brand X, the smallest of the three groups. Borrowing the undecided to build a majority misrepresents the result.
Check that the three groups account for everyone
Why: the percentages must add to 100, or the categories overlap or miss someone.
\[ 25 + 33 + 42 = 100 \quad \checkmark \]
They do, so the 42 percent really are the same teens in both stacked bars. That shared middle group is the whole trick, and the honest report gives all three numbers separately.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 38 — Try It 1.22
Ranking
Six stages of a study with human participants, shuffled. Put them in the order a properly run study goes through them, using the IRB picture and the rest of this part.
Put in order
Arrange the stages from first to last.
Why: The plan comes first, and it fixes the sample size so that nobody can stop at a lucky moment. The board approves it before anyone is recruited. Each participant gives informed consent before being assigned, since they must agree to either treatment. Only then are treatments assigned and data collected, and the study ends by reporting everything, because leaving out unwelcome results distorts the record as surely as inventing data.
The step people most often place wrongly is consent. It has to come before random assignment, because a participant must agree to receive whichever treatment the coin gives them, placebo included.
The first and last steps are the two defences against the quiet fraud of this part. Fixing the sample size in advance blocks the peeking researcher, and reporting everything blocks the researcher who runs five studies and publishes the one that worked.
Socratic
The book ends its discussion of ethics with questions, not answers, and one of them is sharper than it looks. Your doctor takes a blood sample to check your cholesterol. Once it is tested, you expect the rest to be thrown away; at that point it is biological waste.
Discussion prompt
Does a researcher have the right to take that leftover blood and use it in a study without asking you? Argue one side, then give the strongest point for the other side, and say which of the three legal protections is at stake.
Notice that the difficulty is not whether consent matters but when it is needed, which is exactly the question the book poses. Real review boards spend much of their time on cases like this.
The book also points to a website, Retraction Watch, that catalogues research articles withdrawn after being shown to be fraudulent. A quick look shows the problem is larger than most people think, and the defence it recommends is knowledge: learning enough statistics to read a study critically.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — questions of consent and privacy
Section
Part 7
Pattern
Everything in this lesson collapses into one decision and three devices. Follow the flow from the left; the first question decides what kind of conclusion is even available.
Figure (svg): A decision flow. Start: can the researcher assign the explanatory variable? If no, an observational study, which shows association only. If yes, randomly assign units to treatments, then add a control group with a placebo, then blind the subjects and the researchers, arriving at a randomized, controlled, double-blind experiment that can support cause.
When you write a conclusion, carry the reason with it. Because participants were randomly assigned, the difference can be attributed to the treatment. Because people chose for themselves, the difference may be due to a lurking variable such as exercise. The reason is the answer; the verdict alone earns little.
OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 34-38 — the whole section
Check
A quick test of Part 1's vocabulary on a new study. Read carefully: the question asks for one specific part, and three of the options are real parts of the study, just not the one asked for.
\[ \text{120 tomato plants; 40 each get fertilizer A, B, or none; fruit weight measured} \]
Check your understanding
A gardener randomly assigns 120 tomato plants, 40 each, to fertilizer A, fertilizer B, or no fertilizer, and weighs the fruit each plant produces. What are the treatments?
Answer: A
Why: The treatments are the values of the explanatory variable, which here is the fertilizer given. It takes three values, and applying no fertilizer is one of them, because those 40 plants are a group the gardener deliberately set up for comparison.
If you left out the no-fertilizer group, go back to the trap in Part 3. The comparison group is part of the design, and naming it as a treatment keeps you honest about what is being compared.
Ask which of the four answers could be decided before any tomato grows. Only the list of fertilizers can, which is the quickest way to recognise treatments in any study.
Check
This one is about reasoning, not vocabulary. Find the feature of the design that decides the answer before looking at the options.
Check your understanding
A university records that students who use its study centre earn higher grades than students who do not. Which conclusion does this study support?
Answer: A
Why: Students chose for themselves whether to use the centre, so nobody assigned the explanatory variable and this is an observational study. Motivated students may be both more likely to use the centre and more likely to earn good grades, so the most the data support is an association.
The deciding feature was a single phrase: students who use the centre. Whenever the groups are defined by what people chose to do, you are looking at an observational study.
The option about thousands of students is the one to learn from, because it tempts people who understood Part 2 only halfway. Size buys precision, and precision about a biased gap is still a biased gap.
Check
A design question. Identify the specific threat described, then pick the device aimed at that threat, not simply the device that sounds most rigorous.
Check your understanding
In a trial of a new cough syrup, patients are randomly assigned and given identical bottles, but the doctor who scores each patient's cough knows which syrup each patient received. Which change would fix the remaining problem?
Answer: A
Why: Random assignment and identical bottles already handle lurking variables and suggestion. The remaining threat is the doctor's expectation shaping the cough scores, and the device aimed at that is blinding the researcher, which makes the trial double-blind.
Match threat to device every time. The doctor knowing is a knowledge problem, and only a knowledge fix, blinding, touches it.
Notice that every wrong option either does nothing to the real threat or reintroduces a threat that was already solved. A tempting fix that damages the design elsewhere is one of the commonest mistakes in study design.
Explain it to yourself
One argument sits at the centre of this lesson. If you can say it in your own words, you can rebuild the rest of it whenever you need to.
Discussion prompt
In three or four sentences, explain why a randomized experiment can support a claim of cause when an observational study of the same question cannot, even a much larger one. Use a lurking variable in your explanation.
Check your own version for two things. It should say who chooses the treatment in each kind of study, and it should explain why sample size does not rescue the observational one. Those are the two points an examiner looks for.
If your explanation said the groups become identical, soften it. They become alike apart from chance, and chance differences are small and not tilted towards either treatment, which is all the argument needs.
Exit ticket
One design, from the question to the conclusion you would be entitled to write. Use every device from this lesson, and name the threat each one answers.
Discussion prompt
Design a study of whether a magnesium supplement improves sleep in adults. Name the explanatory variable, the treatments and the response; say how you would assign them, what the control group gets, and who is blinded. Then write the conclusion you could draw if the supplement group improved by 12 points more than the control group.
The conclusion is the part to compare most carefully with the model. It names the design features that entitle it to the word attributed, and it states the effect in percentage points, in context.
If your design let volunteers choose their tablets, or gave the control group nothing, go back to the matching part of the lesson. Each of those choices reopens a threat that the design was supposed to close.
Recap
This lesson turned the vague question of whether a study can be trusted into a short list of specific checks, each tied to a specific threat.
| the question | what answers it | what it earns |
|---|---|---|
| what are the parts of the study? | population, sample, units, explanatory variable, treatments, response | a clear statement of what is compared |
| could something else explain the difference? | random assignment spreads lurking variables | the right to say causes |
| could expectation explain it? | a control group given a placebo | the treatment's own effect, by subtraction |
| could anyone's knowledge leak in? | blinding: single for subjects, double for researchers too | a comparison free of hopes |
| can the variable be assigned at all? | if not, an observational study | association only |
| was it conducted honestly? | IRB approval, consent, privacy, a fixed plan | data worth analysing |
\[ \text{random assignment} \Rightarrow \text{cause} \qquad \text{random selection} \Rightarrow \text{generalisation} \]
That line is the lesson's spine: two random steps, two different jobs. A study can earn one without the other, and a good report says which it earned.
The simulations in this deck, the coin splits, the mound of chance gaps and the peeking researchers, are previews. Later chapters measure how wide that mound is and use it to decide when a difference between treatment groups is too large to be chance, which is exactly the question every experiment finally asks.
OpenStax Introductory Business Statistics 2e, §1.4 Experimental Design and Ethics §1.4 — the same design ideas applied to business studies
Want this taught 1-on-1? Alexander tutors Statistics — $55/session, free consultation.