1.4 Experimental Design and Ethics

What a study must do before its results can support a claim about cause: the anatomy of a randomized experiment with its explanatory and response variables, treatments and experimental units; lurking variables and why random assignment rather than random selection is what neutralises them; the control group and the placebo that make the power of suggestion measurable; blinding and double-blinding and what each one protects against; and the studies that cannot be randomized at all, where an observational design shows association and cannot establish cause. Closes with the ethical obligations a researcher carries toward the people being studied.

Subject: Statistics · 65 slides · symbolic lesson

Open the interactive version of this deck

What this lesson covers

The lesson, slide by slide

1. Experimental Design and Ethics

Title

Statistics · Chapter 1 — Sampling and Data

What a study has to do before it may say one thing causes another, and what it owes the people it studies

2. What this lesson gives you

Objectives

Every week a headline says that some food, drug or habit causes something. This lesson gives you the questions that decide whether the study behind the headline could possibly know that, and they turn out to be few and sharp.

The arithmetic in this lesson is light: a few percentages and one subtraction. The weight is in the reasoning, and every conclusion you write should name the feature of the design that entitles you to it. A sentence that says the study proves something, without saying what the design did, is the answer this lesson trains you out of.

Keep one question running under every slide: what else could have produced this difference? A good design is simply a design in which that question has no answer left.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 34-38 — Experimental Design and Ethics

3. Before anything new: two words from Section 1.1

Warm-up

Section 1.1 gave you two words that this lesson leans on in almost every slide: population and sample. Get them straight now, because the first worked example asks for both, and the most common error in this section is mixing them up.

Discussion prompt

A dietitian wants to know whether adults in her city eat enough fibre. She interviews 250 adults at three supermarkets. Name the population and the sample, and say one reason the sample might not look like the population.

Notice the shape of that last answer: it names a specific way the chosen people could differ, and connects it to the thing being measured. That is exactly the move this lesson asks you to make about groups inside a study, not just about the sample as a whole.

Section 1.2 made the same point from the other side: a sample is only as good as the way it was chosen. Here the question moves one step further on. Once the people are chosen, how do you decide who gets which treatment, so that a difference at the end means something?

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.1, p. 6 — population and sample, in the Key Terms

4. The parts of an experiment

Section

Part 1

5. An experiment asks whether one variable changes another

Concept

Does aspirin reduce the risk of heart attack? Is one fertilizer better for roses than another? Each question has the same shape: one variable, which you suspect is doing something, and a second variable, which you suspect it is doing it to.

explanatory and response variables — When one variable causes change in another, the first is the explanatory variable and the affected one is the response variable.

\[ \text{explanatory variable} \;\longrightarrow\; \text{response variable} \]

The arrow is the claim under test, not a fact you already have. In the aspirin question the explanatory variable is the medication and the response variable is whether a heart attack happens. In the roses question it is the fertilizer brand and the growth of the roses.

What makes a study an experiment is who controls the left-hand side. In a randomized experiment the researcher manipulates the values of the explanatory variable, deciding who gets what, and then measures the changes in the response. If the researcher only watches what people already do, it is not an experiment, however careful the watching.

A quick test for which variable is which: ask which one could be set in advance. Nobody can hand out heart attacks, but anyone can hand out pills. The one you could hand out is the explanatory variable.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the purpose of an experiment

6. Treatments are the values; units are what gets measured

Concept

Two more words complete the vocabulary. The explanatory variable takes a few different values in a study, and each value has a name of its own.

treatments — The different values of the explanatory variable. If the explanatory variable is fertilizer brand and two brands are compared, there are two treatments.

experimental unit — A single object or individual to be measured: one person, one rose bush, one plot of soil.

The distinction to hold on to is that a treatment is a value, not a group of people. The group that receives aspirin is a treatment group; aspirin is the treatment. When a question asks for the treatments, it wants the list of values, and a list of groups is half an answer.

\[ \text{explanatory variable: medication} \;\Rightarrow\; \text{treatments: aspirin, placebo} \]

Experimental units are usually easy once you ask what one measurement is taken on. In a drug trial each person gives one yes or no at the end, so each person is a unit. In the roses question, one bush gives one growth measurement, so the bush is the unit, and the gardener is not.

When the units are people they are often called subjects or participants. The words are interchangeable in this course.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — treatments and experimental units

7. Reading the anatomy of an experiment

Notation

Here is the whole vocabulary of Part 1 in one line. Hover or tap each piece to see which word it stands for and the question that finds it in a described study.

Annotate

On: \( \text{units} \xrightarrow{\text{ treatments }} \text{response} \)

  • The individuals or objects measured. Find them by asking what one measurement is taken on.
  • The values of the explanatory variable the researcher hands out. Find them by asking what the groups receive.
  • The researcher assigns the treatments; the units do not choose them. That assignment is what separates an experiment from an observation.
  • The variable measured at the end. Find it by asking what the researcher records to see whether the treatment worked.

The arrow is the part students skip, and it carries the most weight. It says the researcher hands out the treatments. Part 2 will add a single word above it, randomly, and that word is what turns an observed difference into evidence of cause.

Two further words sit around the outside of this line. The population is who the question is about, and the sample is the units actually used. So a complete description of a study names six things: population, sample, units, explanatory variable, treatments and response variable.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 34-35 — the six parts named in Example 1.19

8. The shape every randomized experiment has

Picture it

Before the first example, look at the aspirin study as a picture. The order of the steps is part of the design, and one step in particular is the whole reason the conclusion can be trusted.

Figure (svg): A flow diagram of Example 1.19 from left to right: the population of people aged 50 to 84, a sample of 400 recruited, a random-assignment step that splits them, an aspirin group of 200 and a placebo group of 200, each followed by counting heart attacks, and a final comparison.

Follow the arrows: recruitment decides who is studied, the coin decides who gets what, and only after that is anything measured.

Read it left to right. The question is about people aged 50 to 84; 400 of them are recruited; a random step splits them; each half takes one pill a day for three years; heart attacks are counted in each group; the two counts are compared.

Notice where the randomness sits: after recruitment, before anything is handed out. A study that recruited people at random and then let them choose their own pill would have randomized the wrong step, and Part 2 shows exactly what that costs.

Notice also that the placebo group is drawn as a treatment group, not as an empty box. Both groups swallow a pill every day for three years. The explanatory variable is the pill, and it takes two values.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Example 1.19

9. Example 1.19: the six parts of the aspirin study

Worked example

Researchers want to know whether taking aspirin regularly reduces the risk of heart attack. They recruit 400 people aged 50 to 84, divide them randomly into two groups, and give one group aspirin and the other a placebo, one pill a day for three years. Nobody knows which pill they take. At the end, the researchers count heart attacks in each group.

Population: who the question is about

Why: the question concerns people in this age band, not only the ones who volunteered.

\[ \text{population: people aged 50 to 84} \]

Sample: who was actually studied

Why: the recruited group, all of them, before the split.

\[ \text{sample: the 400 participants} \]

Experimental units: what one measurement is taken on

Why: each person yields one result at the end, heart attack or not.

\[ \text{units: the individual people} \]

Explanatory variable: what the researcher hands out

Why: the pill each person swallows, which the researchers decided.

\[ \text{explanatory variable: oral medication} \]

Treatments: the values it takes

Why: two values, and the placebo is one of them.

\[ \text{treatments: aspirin, placebo} \]

Response variable: what is recorded at the end

Why: a yes or no for each person.

\[ \text{response: whether the person had a heart attack} \]

The response here is categorical, a yes or no per person, which is why the comparison at the end will be a count or a percentage in each group rather than an average.

Check that the parts fit together

Why: the treatments must be values of the explanatory variable, and the response must be something measured on each unit after treatment. Aspirin and placebo are both values of oral medication, and each person is measured once at the end, so all six answers are consistent.

\[ \text{aspirin, placebo} \in \text{oral medication} \quad \checkmark \]

The commonest slip is to answer aspirin for the explanatory variable. Aspirin is one value; the variable is the medication, and it has a second value, the placebo, that is just as much a part of the design.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Example 1.19

10. Match each part of the aspirin study to its answer

Matching

The same study, the other way round. Each description on the right belongs to exactly one part on the left. Do it without looking back, then say to yourself the question that finds each part.

Match the pairs

  • l1. population
  • l2. sample
  • l3. experimental unit
  • l4. treatments
  • l5. response variable
  • r1. people aged 50 to 84
  • r2. the 400 participants
  • r3. one individual person
  • r4. aspirin and a placebo
  • r5. whether a heart attack happened

Why: The population is who the question is about and the sample is who was studied, so the age band and the 400 go there in that order. One person gives one result, so one person is a unit. The treatments are the two pills handed out, and the response is what is recorded at the end.

If you paired the population with the 400, look again at what the question asks. The researchers want to know about aspirin and heart attacks in everyone of that age, and the 400 are the part of that everyone they could reach.

If you paired the units with the 400, the difference is one of scale. The sample is the collection; a unit is one member of it, the thing a single measurement is taken on. The sample is 400 units.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Example 1.19

11. Try It 1.19: three medicines and the height of adults

Worked example

A study looks at the effect of three medicines, A, B and C, on the height of adults aged 30 to 45. Ninety adults are selected randomly and divided into three equal groups; each group takes one medicine for six months, and the average change in height in each group is calculated at the end.

Population and sample

Why: the question is about adults in this age band; ninety were used.

\[ \text{population: adults aged 30 to 45} \qquad \text{sample: the 90 selected} \]

Group sizes

Why: three equal groups share ninety people.

\[ 90 \div 3 = 30 \text{ adults per medicine} \]

Units

Why: each adult gives one change in height.

\[ \text{units: the individual adults} \]

Explanatory variable and treatments

Why: the book says explanatory variables, but there is one variable here with three values.

\[ \text{explanatory: medicine} \qquad \text{treatments: A, B, C} \]

Response variable

Why: measured on each adult, then averaged within each group.

\[ \text{response: change in height over 6 months} \]

Now look at what is missing. There is no placebo group, so every group receives an active medicine. The design can compare A with B with C, but it cannot say whether any of them does anything at all compared with taking nothing, and adults of 30 to 45 are not expected to grow in any case.

Check the parts against each other

Why: A, B and C are the values of the medicine variable, the response is measured on every unit after six months, and three groups of 30 account for all 90 adults.

\[ 30 + 30 + 30 = 90 \quad \checkmark \]

A tidy answer to this Try It also notes the missing control group. Naming the six parts is the start; asking what the design can and cannot compare is the habit.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — Try It 1.19

12. Explanatory or response?

Sorting

Six variables from studies in this section and its exercises. For each one ask a single question: could the researcher set it in advance and hand it out? If so it is explanatory; if it is recorded at the end, it is the response.

Sort into buckets

Sort each variable by the role it plays in its study.

Explanatory: set by the researcher
which pill a participant is given; the scent of the mask worn; texting or not while driving
Response: measured at the end
whether a participant had a heart attack; time taken to finish a maze; seconds to react when the car ahead brakes
ex
Each of these can be decided before the study starts: the researcher chooses the pill, the mask and whether a driver texts. That is what makes them explanatory; their values are the treatments.
re
Each of these is only known after the treatment has had its chance to act: a heart attack, a maze time, a reaction time. They are what the researcher records to see whether the treatment made a difference.

The pattern across the three studies is worth noticing. The thing a study is about, the headline outcome, is almost always the response. The explanatory variable is usually the plainer-sounding thing the researcher controls, which is why students so often name them the wrong way round.

Units of measurement are a useful clue too. A response is often a time, a count or a yes or no recorded per subject, while an explanatory variable is usually a short list of named options.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 35-36 — Examples 1.19 and 1.20, Try It 1.21

13. Lurking variables and random assignment

Section

Part 2

14. Vitamin E takers are healthier. So what?

Concept

Here is the book's example of a study that looks convincing and proves nothing. You recruit a group of people, ask whether they regularly take vitamin E, and find that the ones who do are healthier on average than the ones who do not.

Figure (svg): A causal diagram: a box on the left reading takes vitamin E, a box on the right reading better health, joined by a dashed arrow with a question mark. Above them three boxes, exercise, diet and does not smoke, each with arrows to both the vitamin E box and the health box.

Every lurking variable sends one arrow to who takes the vitamin and one to who is healthy, so it produces the gap without the vitamin doing anything.

The trouble is that the two groups differ in far more than vitamin E. People who take a supplement every day are often the same people who exercise, eat carefully, take other vitamins and choose not to smoke. Any one of those could be making them healthier, with the vitamin along for the ride.

Look at what each yellow box in the picture does. Exercise makes someone more likely to take vitamin E and also makes them healthier, so exercise alone would produce the gap the study saw, even if vitamin E did nothing at all. The same is true of diet and of not smoking.

The researcher here chose nothing: the people chose for themselves whether to take the vitamin. That is the root of the problem, and it is why the study cannot be rescued by measuring more carefully.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the vitamin E study

15. A lurking variable offers a rival explanation

Concept

The yellow boxes have a name. The book defines them in one line, and the definition is worth unpacking because it is stricter than it looks.

lurking variable — An additional variable, not the one being studied, that can cloud a study by offering another explanation for the difference between groups.

To be a rival explanation, a variable has to do two things at once. It has to differ between the groups being compared, and it has to affect the response. A variable that does only one of these clouds nothing.

\[ \text{differs between groups} \;+\; \text{affects the response} \;=\; \text{lurking} \]

Eye colour is a variable, and it certainly differs from person to person, but it does not make anyone healthier, so it cannot explain the vitamin E gap. Exercise passes both tests, which is what makes it dangerous.

To prove that the explanatory variable causes a change, it has to be isolated: the groups must differ in exactly one way, the planned treatment. The rest of this part is about how a design achieves that without knowing what the lurking variables are.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — lurking variables

16. The vitamin E study: finding what else differs

Worked example

Work the vitamin E observation the way you would work any study you are asked to judge: name its parts, then ask whether anything other than the explanatory variable differs between the groups.

Name the explanatory and response variables

Why: the supplement is suspected of causing better health.

\[ \text{explanatory: takes vitamin E} \qquad \text{response: health} \]

Ask who decided the explanatory variable

Why: the participants did, before the study began.

\[ \text{vitamin E chosen by the participant, not assigned} \]

List what else goes with that choice

Why: habits that travel with taking supplements.

\[ \text{exercise, diet, other vitamins, not smoking} \]

Test each against the two conditions

Why: each differs between takers and non-takers, and each affects health.

\[ \text{differs between groups} + \text{affects health} \Rightarrow \text{lurking} \]

State the conclusion the design allows

Why: an association, with rival explanations still standing.

\[ \text{takers healthier: association, not cause} \]

That conclusion does not say vitamin E is useless. It says this study cannot tell. The vitamin might help, might do nothing, or might even do slight harm that the takers' other habits more than make up for; all three are consistent with what was observed.

Check the conclusion by imagining vitamin E does nothing

Why: if the gap would still appear with a useless vitamin, the study cannot be evidence for it. Exercisers take the vitamin more and are healthier anyway, so a useless vitamin would still show a gap.

\[ \text{useless vitamin} \Rightarrow \text{gap still appears} \Rightarrow \text{no evidence} \]

That imagined check is the most useful test in this lesson. Ask what the data would look like if the treatment did nothing; if the answer is the same as what was seen, the study has shown nothing about the treatment.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the vitamin E study

17. Random assignment: let a coin choose

Concept

The cure is to take the choice away from the subjects. If a coin decides who gets vitamin E, then exercise, diet and smoking have no way to influence who ends up in which group.

Figure (svg): Four dot grids of the same 400 simulated people, with regular exercisers drawn in yellow and sorted to the front. Top row, people who chose vitamin E for themselves: 117 of 182 takers exercise but only 43 of 218 non-takers. Bottom row, people split by a coin: 83 of 200 in one group and 77 of 200 in the other.

Same 400 people both times. Letting people choose piles the exercisers into one group; letting a coin choose leaves them almost evenly split.

These are the same 400 simulated people twice, with the regular exercisers in yellow. In the top row, people chose for themselves, and the exercisers flocked to the vitamin: 117 of the 182 takers exercise, against only 43 of the 218 non-takers. That is 64.3 percent against 19.7 percent.

In the bottom row a coin decided. The exercisers are now split 83 and 77, or 41.5 percent against 38.5 percent. The coin knew nothing about exercise, so exercise could not steer anyone into either group.

This is what the book means by saying that when subjects are assigned treatments randomly, all the potential lurking variables are spread equally among the groups. It works for exercise, and it works in exactly the same way for every variable nobody thought to measure, because the coin ignores all of them at once.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — random assignment spreads lurking variables among the groups

18. How big a gap does the coin leave?

Worked example

The two coin groups were not identical: one had 83 exercisers and the other 77. Work out how big that leftover gap is, and compare it with the gap self-selection produced, using the counts from the dot grids.

Self-selected: exerciser share among the takers

Why: 117 of the 182 who chose vitamin E.

\[ \dfrac{117}{182} = 0.643 = 64.3\% \]

Self-selected: share among the non-takers

Why: 43 of the 218 who did not.

\[ \dfrac{43}{218} = 0.197 = 19.7\% \]

The self-selected gap

Why: subtract the shares, unrounded, and round at the end.

\[ 64.29 - 19.72 = 44.6 \text{ percentage points} \]

Coin-assigned: the two shares

Why: 83 and 77 exercisers, 200 people in each group.

\[ \dfrac{83}{200} = 41.5\% \qquad \dfrac{77}{200} = 38.5\% \]

The coin-assigned gap

Why: a simple subtraction this time, since both groups have 200 people.

\[ 41.5 - 38.5 = 3.0 \text{ percentage points} \]

So the coin left a gap of 3 points and self-selection left one of 44.6. Random assignment does not make the groups identical; it makes the leftover difference small and, just as important, not tilted towards either treatment.

Check the counts against the whole

Why: the exercisers in each pair of groups must add back to the 160 exercisers among the 400.

\[ 117 + 43 = 160 \qquad 83 + 77 = 160 \quad \checkmark \]

Both pairs account for all 160 exercisers, so the two splits really are of the same 400 people, and the only thing that changed between them is who made the choice.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — random assignment and lurking variables

19. A thousand coin tosses, and one self-selected gap

Picture it

One coin split left a gap of 3 points. Was that luck? Re-run the coin a thousand times on the same 400 people and record the gap in exerciser share each time.

Figure (svg): A histogram of the gap in exerciser share between two coin-assigned groups over 1,000 re-draws: a mound centred on zero running from about minus 18 to plus 16 points. Far to the right, a dashed line at 44.6 points marks the gap produced by self-selection.

A coin can be unlucky, but only by a few points; not one of the 1,000 re-draws comes anywhere near the gap that self-selection produced.

The re-draws pile up around zero. In this run, 734 of the 1,000 left a gap of 5 points or less and 963 left a gap of 10 or less; the most lopsided split of all was 18 points. That is how big chance imbalances get with 200 people a group.

Now look at the dashed red line at 44.6, where self-selection put the gap. Not one of the thousand coin splits gets within 25 points of it. Self-selection does not produce an unlucky split; it produces a split of a completely different kind.

This picture is the seed of most of the rest of the course. Later chapters measure exactly how wide this mound is, and use it to decide whether a difference between treatment groups is bigger than the coin alone would produce.

20. How much chance imbalance is left, for groups of any size?

Tweak it

The mound on the last slide had a typical width of about 5 points for groups of 200 people. The formula in the readout gives that typical width for any group size and any share of people with the lurking trait; you do not need to derive it, only to read it.

Parameter explorer

Drag the share p of people with the trait, then read the curve: how large is the typical chance gap for groups of 200, and for groups of 20?

\[ \text{typical gap} = 100\sqrt{\dfrac{2 \cdot {p}(1 - {p})}{n}} \text{ points} \]

  • p — from 0.05 to 0.5: share with the trait, p

At p of 0.4 and groups of 200 the formula gives 4.9 points, which matches the thousand re-draws, whose spread came out at 4.95. The formula and the simulation are two views of the same coin.

Now slide along the curve. With groups of 20 the typical gap is 15.5 points, big enough to matter; with groups of 1,000 it is 2.2. Random assignment balances lurking variables better the more units it has to work with, which is one honest reason experiments want many subjects.

Keep this separate from what comes next. Here, a bigger experiment shrinks a gap that was already centred on zero. On the next slide, a bigger observational study shrinks nothing that matters, because its gap was never centred on zero.

21. A bigger sample does not cure a lurking variable

Concept

A tempting reply to the vitamin E problem is to survey more people. Surely forty thousand people outweigh a few lurking variables? The simulation below says no, and says why.

Figure (svg): Dots for simulated studies of sizes 100 to 40,000 on a log scale. Orange dots, observational studies of people who chose vitamin E, scatter widely at first and settle on a dashed line at 11.6 points. Blue dots, randomized experiments, settle on the zero line. In the simulation vitamin E has no effect at all.

More people make both kinds of study steadier. Only one of them is steady around the truth; the other is steady around the lurking variable's effect.

In this simulated world vitamin E does nothing at all; health depends only on exercise. Each orange dot is an observational study of people who chose for themselves, each blue dot a randomized experiment, at sizes from 100 to 40,000.

Small studies of either kind scatter widely. As they grow, both kinds steady down, but to different places. The randomized studies settle on zero, the truth. The observational studies settle on 11.6 points, the gap exercise creates: at 40,000 people the orange dot still sits at 12.7 and the blue one at 0.3.

So size buys precision, not honesty. A large observational study measures the association very precisely, and the association includes every lurking variable. Only the coin removes them, which is why the book says an experiment isolates the explanatory variable by random assignment and not by numbers.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — isolating the explanatory variable

22. Double the survey: what happens to the gap?

Prediction

Commit to an answer before reasoning it through; the last slide gave you everything you need, but it is worth finding out whether the idea has landed.

Predict first

An observational survey of 5,000 people finds vitamin E takers 12 points healthier. The researchers repeat it with 10,000 people chosen the same way. What should you expect?

  • A gap near 12 points again, measured a little more precisely
  • A gap closer to zero, because a bigger sample cancels the lurking variables
  • A gap about twice as large, because twice as many people were surveyed
  • No gap, because a large enough survey counts as an experiment

Correct: A gap near 12 points again, measured a little more precisely

Why: Doubling the sample does nothing to who chose the vitamin, so the exercisers still crowd into the taker group and the lurking gap is still there. A bigger sample only pins that same gap down more precisely.

If you chose the answer that says the gap shrinks towards zero, you have mixed up the two effects of size. A bigger sample shrinks random scatter, the wobble between one study and the next. It does not shrink a gap that is built into the way the groups were formed.

The answer that says the gap doubles confuses a count with a percentage. The gap is a difference in percentages, and percentages do not grow because the sample does.

The last option is the one to reject most firmly. What makes a study an experiment is who assigns the treatment, never how many people take part. A survey of a million volunteers who chose their own vitamins is still an observational study.

23. Random selection or random assignment?

Discrimination

Two different random steps can appear in a study, and they do different jobs. For each description, ask what the randomness decided: who is in the study, or what they receive once they are in it.

Sort into buckets

Sort each description by the job its randomness does.

Random selection: who is studied
1,000 adults drawn at random from the city's voter list; 30 schools picked by lottery from every school in the state; a phone survey dials numbers generated at random
Random assignment: who gets which treatment
each participant's group decided by a coin flip; which half of the maze trials carried the floral scent, decided at random; a random-number table decides which rose bushes get brand A
sel
The voter list, the school lottery and the random dialling all decide who ends up in the study. That makes the sample representative of the population, which is what lets you generalise, but it says nothing about who receives what.
asg
The coin, the random order of the mask trials and the random-number table all decide which treatment a unit receives after it is already in the study. That is what spreads lurking variables evenly and lets a difference be read as cause.

A study can have either kind of randomness, both, or neither. The aspirin study used random assignment but recruited volunteers, so its conclusion about cause is strong while its reach to all people aged 50 to 84 depends on how typical the volunteers were.

Keep the two jobs in one sentence each. Random selection earns generalisation: the sample looks like the population. Random assignment earns causation: the groups look like each other.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.2, p. 18 — random sampling

24. Trap: a random sample does not make an experiment

Trap

The trap

Here is a conclusion that sounds careful and is wrong. The survey really did use a random sample, which is exactly why the error is so persuasive.

\[ \text{random sample of adults} \]

\[ \text{takers are healthier} \]

\[ \Rightarrow \text{ vitamin E works} \]

The randomness in this study decided who was surveyed. It did not decide who takes vitamin E; the adults decided that themselves, long before anybody called.

The fix

Name the randomness correctly, and the conclusion shrinks to what the design supports.

\[ \text{random selection:} \]

\[ \text{sample represents adults} \]

\[ \text{self-chosen vitamin E:} \]

\[ \text{association only} \]

A random sample is valuable: it means the association is probably true of adults in general, not just of an odd group. But the association is still an association, with exercise, diet and smoking tangled up in it.

To earn the word causes, the researcher must be the one who decides who takes vitamin E, and must decide it at random. When you read a study, find the random step and ask what it decided.

25. Break this claim: breakfast prevents weight gain

Counterexample

A study reports that people who eat breakfast weigh less, on average, than people who skip it, and concludes that eating breakfast prevents weight gain. The observation may be perfectly accurate. Your job is to break the conclusion.

Discussion prompt

Name two lurking variables that could produce this gap even if breakfast did nothing, showing for each that it differs between the groups and affects weight. Then describe the study that would settle the question.

Notice that each lurking variable needed both halves of the test. A variable that only affects weight, or only affects who eats breakfast, cannot produce the gap on its own.

Notice too that you did not need to know which lurking variable was really at work. That is the power of the coin: the settling study protects against the rivals you named and the ones you never thought of, all at once.

26. The control group and the placebo

Section

Part 3

27. Believing you took it can work almost as well as taking it

Concept

Random assignment handles the differences people bring with them. There is a second problem it does not handle: what happens to people because they know they are in a study. The book calls it the power of suggestion, and quotes a study of a performance-enhancing substance.

Figure (svg): A two by two grid. Columns: given the real substance, drawn as a filled pill, and given an inert pill, drawn as a hollow pill. Rows: told they took the substance, and told they did not. The two cells where subjects believed they had taken it are shaded bright and marked fast; the cell given the drug without knowing is marked no significant gain.

Read along the top row: believing you took it was nearly as good as taking it. Read down the left column: taking it without believing did nothing measurable.

The study crossed two things: what athletes actually received, and what they were told. Those who believed they had taken the substance ran almost as fast as those who really had it. Those who got the substance without knowing showed no significant gain.

Read the grid along the top row and then down the left column. Along the top, belief does nearly all the work. Down the left, the substance without belief does almost nothing. In a study where everyone who got the drug also knew it, the two would be completely tangled.

So a treatment group that knows it is treated is measuring two things at once: the treatment, and the effect of expecting the treatment to work. Part 3 is about separating them.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 34 — the power of suggestion, McClung and Collins 2007

28. The control group gets a placebo, and a placebo is a treatment

Concept

To measure the power of suggestion, researchers set one group aside as a control group and give it a placebo. The book's definition of a placebo is precise, and one word in it surprises people.

placebo — An active treatment that cannot directly influence the response variable: a sugar pill, a saline injection, a sham procedure.

Active, because the control group really does something: swallows a pill every day, visits the clinic, fills in the forms. Everything about being in the study happens to them. Only the ingredient that could act on the response is missing.

\[ \text{treatment group: study + ingredient} \qquad \text{control: study only} \]

That is exactly why the control group is useful. It experiences everything the treatment group does except the one thing under test, so the difference between the groups is the effect of that one thing. The book puts it as balancing the effects of being in an experiment against the effects of the active treatment.

A control group that got nothing at all, no pill and no visits, would not do this job. It would differ from the treatment group in two ways, the ingredient and the whole experience of being treated, and a difference at the end could come from either.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — the control group and the placebo

29. What the difference between the arms measures

Notation

Here is the subtraction every placebo-controlled study ends with. Tap each piece to see what it contains; the point is that one ingredient appears on both sides and cancels.

Annotate

On: \( \underbrace{55\%}_{\text{treated}} - \underbrace{42\%}_{\text{placebo}} = 13 \text{ points} \)

  • Improvement from the ingredient plus improvement from being in the study and expecting help.
  • Improvement from being in the study and expecting help, and nothing else, because the pill cannot act on the response.
  • The shared part cancels. What is left is the ingredient's own effect.
  • A difference of two percentages is measured in percentage points, not percent.

Written out, the argument is one line of algebra. If both arms share the same study effect, subtracting one arm from the other removes it.

\[ (\text{drug} + \text{study}) - (\text{study}) = \text{drug} \]

That cancellation only happens if the study effect really is the same in both arms. It is the same only if both arms believe the same thing about what they are taking, which is where blinding, in Part 4, comes in.

30. Reading a placebo-controlled result: 55 against 42

Worked example

A sleep supplement is tested against a placebo. Participants are randomly assigned, and neither they nor the staff know who has which pill. After four weeks, 55 percent of the supplement arm and 42 percent of the placebo arm report sleeping better. Report the supplement's effect.

Figure (svg): Two bars: a placebo arm at 42 percent reporting better sleep and a supplement arm at 55 percent. A dashed line at 42 runs across both bars; the part of the supplement bar above it, 13 points, is highlighted as the supplement's own effect.

Both arms share the same 42 points of improvement from being in a study. Only the orange cap belongs to the supplement.

Name what the placebo arm shows

Why: 42 percent improved on a pill that cannot act on sleep: the effect of being in the study.

\[ \text{placebo arm: } 42\% \]

Subtract to isolate the supplement

Why: the study effect is common to both arms, so it cancels.

\[ 55\% - 42\% = 13 \text{ percentage points} \]

Say it as a relative change too

Why: 13 points on top of a base of 42.

\[ \dfrac{13}{42} = 0.3095 \approx 31\% \text{ more people improving} \]

Both numbers are correct, and they answer different questions. Thirteen points is the absolute effect: out of every hundred people, thirteen more improved. Thirty-one percent is the relative effect, and it sounds larger, which is why advertisements prefer it.

Write the conclusion in context

Why: the random assignment and the placebo together license a causal sentence.

\[ \text{the supplement raised the share sleeping better by 13 points} \]

What you must not write is that the supplement helped 55 percent of people. Forty-two of those 55 points would have happened with a sugar pill.

Check by adding the pieces back

Why: the placebo arm's rate plus the supplement's effect should rebuild the treated arm's rate.

\[ 42 + 13 = 55 \quad \checkmark \]

The subtraction and the addition agree, so the 13 points is exactly the part of the treated arm the placebo arm cannot account for.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — the control group balances the effect of being in an experiment

31. Without the placebo arm, what would the study have claimed?

Estimation

Suppose the same trial had been run with no control group at all: everyone took the supplement, and 55 percent reported sleeping better. Estimate what that trial would have announced, compared with the truth from the last slide.

Predict first

With no placebo arm, how would the reported effect compare with the supplement's real effect of 13 points?

  • It would overstate the effect several times over
  • It would be about right, since 55 percent still improved
  • It would understate the effect
  • It would be impossible to say anything at all

Correct: It would overstate the effect several times over

Why: With nobody to compare against, all 55 percent of the improvement looks like the supplement's doing, while the placebo arm showed that 42 of those points happen with a sugar pill. The real effect is 13 points, so the uncontrolled trial credits the supplement with more than four times its share.

The ratio is worth computing: 55 divided by 13 is about 4.2. An uncontrolled trial of this supplement would have credited it with more than four times the effect it really has, and it would have looked like strong evidence.

This is why the words placebo-controlled matter on a drug label. An improvement rate on its own tells you what happened to the treated people, not what the treatment did to them.

The other options each miss the point in a different way. The uncontrolled trial does say something, just the wrong thing. And it cannot understate, because it adds the study effect to the drug effect instead of removing it.

32. Trap: calling the control group untreated

Trap

The trap

Asked for the treatments in the aspirin study, many students write one treatment and one empty group.

\[ \text{treatment: aspirin} \]

\[ \text{control group: untreated} \]

It sounds natural, because the control group receives no aspirin. But it misdescribes what the control group experiences, and it misleads you about what the comparison measures.

The fix

The control group takes a pill every day for three years, exactly as the aspirin group does. That pill is a treatment.

\[ \text{explanatory: oral medication} \]

\[ \text{treatments: aspirin, placebo} \]

Getting this right matters because the whole value of the control group lies in how much it shares with the treatment group. Describe it as untreated and you invite the design error that comes with the word: a control group given nothing at all, which cannot separate the ingredient from the experience of being treated.

The book's wording is the one to keep: the control group is given a placebo treatment, an active treatment that cannot directly influence the response variable.

33. Try It 1.20: measuring the placebo effect itself

Worked example

The Placebo Research Group wants to find the extent of placebo effects. Randomly selected men take a test, take a pill that induces a mild headache, and take the test again. For half of the men, chosen at random, the pill is replaced by a similar pill with no effect. The researchers record the change in test time.

Explanatory variable

Why: what the researchers decide for each man.

\[ \text{explanatory: which pill is taken} \]

Response variable

Why: measured on each man, before and after.

\[ \text{response: change in time to complete the test} \]

Treatments

Why: the two values of the pill variable.

\[ \text{treatments: headache pill, look-alike inert pill} \]

Lurking variables

Why: the pill was assigned at random, so habits, health and test skill are spread across both halves.

\[ \text{random assignment} \Rightarrow \text{no systematic lurking variable} \]

One thing does still affect both groups: every man takes the test twice, and people usually do better the second time. That practice effect is real, but it is the same in both halves, so it cancels when the halves are compared, exactly as the study effect did on the last worked example.

Blinding

Why: the pills look alike, so the men need not know which they had, and neither need the researchers timing them.

\[ \text{subjects and timers can both be blinded} \]

There is a catch worth raising. A man who develops a headache may guess which pill he had, and that guess could leak back into his performance. A good design would ask afterwards which pill each man thought he got, to see whether the blinding held.

Check the design against its purpose

Why: to measure the placebo effect, the comparison must differ only in the pill's active ingredient. Random assignment and look-alike pills make that the only planned difference.

\[ \text{only planned difference: the active ingredient} \quad \checkmark \]

Notice how the pieces line up with the goal. The design is aimed at the placebo effect itself, so the inert pill here is not a background control; it is half of the thing being studied.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Try It 1.20

34. Blinding: who is allowed to know

Section

Part 4

35. A placebo only works if nobody knows it is one

Concept

A placebo has a weakness built into it. If you are in a study and you know your pill contains no medicine, the power of suggestion is no longer at work on you, and the control group stops measuring what it was meant to measure.

blinding (masking) — A person involved in a study is blinded when they do not know who is receiving the active treatment and who the placebo.

Blinding the subjects is what makes the cancellation in Part 3 legitimate. Both arms believe the same thing, that they might be getting the real treatment, so both arms carry the same expectation, and subtracting one from the other removes it.

\[ \text{blinded subjects} \Rightarrow \text{same expectation in both arms} \]

The book puts this neatly: blinding preserves the power of suggestion. It does not remove it; it makes sure it acts equally on everybody, so that it can be subtracted away.

A study in which only the subjects are blinded is called single-blind. That protects against the subjects' expectations, but a study has other people in it who might also expect something.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — blinding or masking

36. Double-blind: the researchers must not know either

Concept

The researchers who work with the subjects have expectations too. A researcher who knows which patients are on the new drug may, without meaning to, encourage them more, record a borderline symptom more kindly, or time a maze with a slightly quicker thumb.

Figure (svg): A grid of eyes. Three rows: open study, single-blind, double-blind. Two columns: the subject and the researcher who works with the subject. In the open study both eyes are open; in single-blind the subject's eye is shut and the researcher's open; in double-blind both are shut.

Blinding is about knowledge, not eyesight: a shut eye means that person cannot tell which treatment any subject received.

Read the grid by rows. In an open study everyone can see who got what. In a single-blind study the subject cannot. In a double-blind study neither the subjects nor the researchers working with them can, and the book defines it exactly that way.

double-blind experiment — An experiment in which both the subjects and the researchers involved with the subjects are blinded.

Each shut eye closes one route by which expectation could leak into the response. Blinding the subject protects against the power of suggestion. Blinding the researcher protects against the researcher's hopes shaping what gets measured or how.

Someone has to hold the key, of course, or the results could never be sorted. The allocation list is kept by a person who has no contact with the subjects, and it is opened only when the measurements are complete.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 35 — double-blind experiments

37. Example 1.20: scented masks and maze times

Worked example

A foundation studies whether smell affects learning. Each subject completes pencil and paper mazes six times while wearing a mask: three times with a floral-scented mask and three times with an unscented one. Whether the floral mask comes in the first three trials or the last three is decided at random. Researchers record each maze time and the subject's impression of the scent.

Figure (svg): A design diagram for the mask study. All subjects enter a random-assignment box, which splits them into two sequences of six maze trials. Sequence one wears the floral mask for trials 1 to 3 and the unscented mask for 4 to 6; sequence two does the reverse.

Each subject is compared with themself, and the coin decides only when the scent comes, so that order cannot favour either mask.

(a) Explanatory and response variables

Why: the scent is set by the researchers; the maze time is recorded.

\[ \text{explanatory: scent} \qquad \text{response: time to finish the maze} \]

(b) Treatments

Why: the two values of scent.

\[ \text{treatments: floral mask, unscented mask} \]

(c) Lurking variables

Why: every subject receives both treatments, so the groups cannot differ in who is in them.

\[ \text{same people under both masks} \]

That is a different design from the aspirin study. Instead of comparing two groups of people, each person is compared with themself, so differences in skill, age or eyesight cannot favour either mask. What is randomized is the order, and the next slide shows why the order needed randomizing.

(d) Can the study be blinded?

Why: the subjects can smell the flowers, so they cannot be blinded; the researcher timing the maze need not know which mask is on.

\[ \text{subjects: no} \qquad \text{timing researcher: yes} \]

Check that each answer matches the design

Why: scent is what the researchers change between trials, maze time is measured on every trial, and the random order is the only random step, so it is the one that protects against lurking variables.

\[ \text{randomized: the order} \Rightarrow \text{practice cannot favour a mask} \quad \checkmark \]

The subjects' impression of the scent was recorded as well, but it is a second response, not a treatment: nobody assigned it.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 35-36 — Example 1.20

38. Why the order of the masks had to be random

Picture it

Suppose the designers had not randomized, and every subject wore the floral mask for the last three trials. The curve below is an illustration of a practice effect, with the scent doing nothing at all.

Figure (svg): An illustrative practice curve: maze time falling from 70 seconds on trial 1 to about 42 on trial 6, with dots at each trial. Trials 4 to 6 are shaded pink as if the floral mask were always worn last; their average, 44.4 seconds, sits well below the 59.7-second average of trials 1 to 3.

People get faster with practice whatever they wear. A scent that always came last would collect about 15 seconds of credit it did not earn.

People get faster at mazes with practice. On this curve the first three trials average 59.7 seconds and the last three average 44.4 seconds, a gap of about 15 seconds that comes entirely from practice.

\[ 59.74 - 44.41 = 15.3 \text{ seconds} \]

If the floral mask always came last, all 15 of those seconds would be credited to the scent. Practice would be a lurking variable in the purest form: it differs between the floral and unscented trials, and it affects maze time.

Randomizing the order breaks that link. Half the subjects meet the scent early and half meet it late, so the practice advantage lands equally on both masks and cancels in the comparison. The book's answer, that random assignment eliminates the problem of lurking variables, is this picture in one sentence.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Example 1.20, part c

39. Each design device, and the threat it answers

Comparison

Four devices have appeared so far, plus the random sampling of Section 1.2. Each exists to answer one specific threat to a study's conclusion. Fill in the threat for each one; one word or a short phrase is enough.

Comparison matrix

design devicewhat it doesthe threat it answers
random assignmenta coin decides who gets which treatmentlurking variables
placebo for the control groupboth groups have the full study experiencepower of suggestion
blinding the subjectssubjects do not know which they gotsubjects' expectations differing between groups
blinding the researchersstaff do not know who got whatresearcher bias
random selectionchance decides who is in the sampleunrepresentative sample

Read down the last column and you have the checklist for judging any study. When someone tells you a result, go through it: were there lurking variables, could suggestion explain it, could the staff have nudged it, and does the sample represent anyone beyond itself?

Notice which row is different in kind. The first four protect the comparison inside the study, which is what the claim of cause rests on. Random selection protects the reach of the result outside the study, which is what generalisation rests on.

A study can pass the first four and fail the last, like a well-run trial on volunteers from one hospital. It then shows cause convincingly, for people like its volunteers.

40. Try It 1.21: texting and reaction time, designed from scratch

Worked example

You want to know how texting affects driving. Design a study of how many seconds a driver takes to respond when the car ahead brakes, while texting and while driving only.

(a) Explanatory and response variables

Why: texting is what you control; the reaction time is what you record.

\[ \text{explanatory: texting or not} \qquad \text{response: reaction time (s)} \]

(b) Treatments

Why: the two values of the explanatory variable.

\[ \text{treatments: text and drive, drive only} \]

(c) Choosing participants

Why: licensed drivers who already text, over a range of ages, tested in a driving simulator so nobody is put in danger.

\[ \text{licensed drivers who text; simulator only} \]

Figure (svg): Two design panels. Left, two separate groups: eight drivers who only drive, eight others who text, each drawn as a person. Right, the same eight drivers each tested twice, once texting and once not, with each person's two sessions joined and a coin deciding which comes first.

Comparing each driver with themself removes the biggest source of variation, the differences between people; randomizing the order handles practice.

(d) Two random groups: good or not?

Why: it works, but people's reaction speeds differ so much that the difference between people can swamp the difference texting makes; testing each driver under both conditions removes it.

\[ \text{better: each driver does both, order random} \]

(e) Lurking variables

Why: whatever differs between the conditions and affects reaction time.

\[ \text{experience, age, fatigue, practice, simulator familiarity} \]

With every driver doing both, experience and age are identical across the two conditions, because they are the same people. Practice and fatigue remain, because the second session comes after the first, and randomizing which session comes first spreads them evenly, exactly as in the mask study.

(f) Blinding

Why: drivers know when they are texting, so they cannot be blinded; the person scoring the reaction times can be, if the recordings are unlabelled.

\[ \text{drivers: no} \qquad \text{scorer: yes} \]

Check the design against the lurking list

Why: each variable named in (e) must be handled by something in the design. Same drivers in both conditions handles experience and age; the random order handles practice and fatigue; the simulator is the same for everyone.

\[ \text{every lurking variable matched to a device} \quad \checkmark \]

That is the discipline of design in one move: list the rivals, then point to the feature of the study that disarms each one. Any rival with nothing pointing at it is a hole.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Try It 1.21

41. Find the flaw in this trial design

Error analysis

A clinic plans to test a new pain cream. Read the plan, then tap each part of the line below to see where it fails and which device from the last few slides would fix it.

\[ \text{Patients choose cream or no cream; the doctor rates pain} \]

Annotate

On: \( \text{choose} \;\to\; \text{no cream} \;\to\; \text{doctor rates} \)

  • Self-selection: the patients most bothered by pain, or most hopeful, may pick the cream. Fix with random assignment.
  • The comparison group gets nothing, so the experience of treatment is not balanced. Fix with a placebo cream that looks and smells the same.
  • The doctor knows who has the cream and may rate those patients more hopefully. Fix by blinding the doctor, which with a placebo cream makes the trial double-blind.

Every part of the plan fails, and each failure is a different one of the threats you have met. That is common: a design put together without the checklist usually trips on all of them together.

The repaired plan uses the same words the book uses for the aspirin study: patients randomly assigned to the new cream or an identical placebo cream, with neither patients nor the rating doctor knowing which. Randomized, placebo-controlled, double-blind; each adjective answers one of the three failures.

Pain is especially exposed to all three failures because it is reported by the patient and judged by the doctor. When a response is a judgement rather than a measurement, blinding matters most.

42. Before you design: which people could possibly be blinded?

Step zero

Before choosing a design, it pays to ask which blinding is even possible, because some treatments announce themselves. Decide this first, and the rest of the design follows from it.

Discussion prompt

For each proposed study, say whether the subjects can be blinded and whether the people measuring the response can be: (1) a new blood-pressure tablet against a look-alike placebo; (2) a yoga class against no class, with flexibility measured; (3) surgery against a sham incision, with recovery rated by a nurse.

The pattern is the same as in the mask study. When the subjects cannot be blinded, you can nearly always still blind the person who measures, and that is worth doing because it removes one of the two routes by which expectation leaks in.

The surgery case also shows that blinding is a choice about roles. The person who must know, the surgeon, is kept away from the measurement, and the person who measures is kept away from the knowledge.

43. When you cannot randomize

Section

Part 5

44. Some explanatory variables cannot be handed out

Concept

Everything so far has assumed the researcher can decide who gets which treatment. For many of the questions people care most about, that is impossible, or it would be wrong to try.

Nobody can assign a person's birth order, sex or childhood. Nobody may ethically assign people to smoke for twenty years to see what happens to their lungs. In these cases the researcher can only observe the values people already have, and the study is an observational study.

\[ \text{cannot assign} \Rightarrow \text{observational study} \Rightarrow \text{association} \]

An observational study is not worthless. It can show that two variables go together, that smokers have more lung cancer, that firstborns score differently on some test, and it can measure how strongly. What it cannot do on its own is rule out the rival explanations, because nothing was done to spread the lurking variables across the groups.

When a question can only be studied by observation, the honest word for the result is association. Stronger claims need the kind of evidence that piles up from many observational studies pointing the same way, with the plausible lurking variables measured and accounted for, as happened with smoking.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Example 1.21

45. Example 1.21: birth order and personality

Worked example

A researcher wants to study the effect of birth order on personality. Explain why this study could not be conducted as a randomized experiment, and name the main problem with a study that cannot be.

Figure (svg): A family timeline. Three children are born when the parents are 25, 29 and 34, marked firstborn, middle and last-born. Under each child, what else differs: parents' age, number of older siblings, and the household's circumstances at that time. A red crossed-out coin shows that birth order cannot be assigned.

Everything in yellow comes packaged with birth order, so a personality difference between firstborns and last-borns could belong to any of it.

Identify the explanatory variable

Why: birth order is what is suspected of shaping personality.

\[ \text{explanatory: birth order} \qquad \text{response: personality} \]

Ask whether it can be assigned

Why: no coin can decide who is born first; people arrive with their birth order.

\[ \text{birth order cannot be randomly assigned} \]

Name what is lost

Why: random assignment is what spreads lurking variables evenly, and without it they stay put.

\[ \text{no random assignment} \Rightarrow \text{lurking variables remain} \]

Name some of them

Why: the timeline shows what travels with birth order.

\[ \text{parents' age, number of siblings, family income at the time} \]

Each of those passes both halves of the lurking test. A firstborn has younger, less experienced parents and no older siblings; a last-born has neither of those. Any of these differences could shape personality, and all of them come packaged with birth order.

Check the answer against the book's principle

Why: when subjects cannot be assigned to groups at random, the groups differ in more than the explanatory variable, which is exactly the problem named above.

\[ \text{groups differ in more than birth order} \quad \checkmark \]

So the most such a study can report is that personality differs with birth order, an association, and a careful report lists the lurking variables it could not rule out.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 36 — Example 1.21

46. Could this question be answered by an experiment?

Discrimination

Six research questions. For each, decide whether a researcher could randomly assign the explanatory variable, both practically and ethically. If not, the question can only be studied by observation.

Sort into buckets

Sort each question by the kind of study that can answer it.

An experiment is possible
Does a new fertilizer make roses grow taller?; Does a reminder text raise attendance at clinic appointments?; Does playing music in a shop change how long customers stay?
Only an observational study
Does being an only child affect confidence?; Does heavy drinking in pregnancy affect birth weight?; Do people who live near a motorway have more asthma?
exp
Fertilizer, reminder texts and shop music can all be handed out by a coin, cheaply and without harm, so each question can be settled by a randomized experiment.
obs
Nobody can assign whether a child has siblings, nobody may assign drinking in pregnancy, and nobody can make people move house next to a motorway. These can only be observed, so each study shows association and has to wrestle with lurking variables.

Two different reasons land questions in the second group. For sibling status and home address the obstacle is practical: the variable cannot be assigned. For drinking in pregnancy the obstacle is ethical: it could be assigned, but assigning a likely harm is forbidden.

The ethical case links to Part 6. The rules that protect subjects are one of the main reasons that so much of what we know about health comes from observational studies, and why those studies have to be read so carefully.

47. Trap: reading an observational headline as a cause

Trap

The trap

A news story reports an observational study of 20,000 adults with the headline below. The numbers in it are genuine; the conclusion is the problem.

\[ \text{coffee drinkers live longer} \]

\[ \Rightarrow \text{ drink coffee to live longer} \]

The step from the first line to the second assumes that coffee is the only difference between drinkers and non-drinkers.

The fix

Nobody assigned the coffee. Ask what else differs between people who choose to drink it and people who do not.

\[ \text{drinkers differ in} \]

\[ \text{income, health, habits} \]

\[ \text{observational study:} \]

\[ \text{association only} \]

People who are already ill often give up coffee, which alone would make the remaining drinkers look healthier. Income and work patterns travel with coffee too. Any of these could produce the gap.

The honest sentence is that in this study coffee drinkers lived longer on average, and the study cannot tell whether coffee is the reason. Twenty thousand people make that association precise; they do not make it causal.

48. A health claim from this week's news

Real world

This is the skill the lesson exists for, used on the world rather than the textbook. Find a real health or lifestyle claim in the news this week and look up how the study behind it was done.

Discussion prompt

Say whether the study assigned the explanatory variable. If it did, say whether it used a placebo and blinding; if it did not, name one plausible lurking variable. Then write the one sentence you would use to report the finding honestly.

Look for the telltale words as you read. Linked to, associated with and tied to usually signal an observational study; phrases such as randomly assigned or placebo-controlled trial signal an experiment.

Notice too how often the headline is stronger than the study. The research article may say associated with while the headline says causes. When the two disagree, trust the design, not the headline.

49. Ethics: what a study owes

Section

Part 6

50. Numbers do not lie, but people using them can

Concept

The second half of the section turns from how a study should be designed to how it should be conducted. The book's central case is the social psychologist Diederik Stapel, a professor at Tilburg University, whose fraud led to retractions from some of the world's leading journals.

An investigation across three universities found falsified data in over 55 of his papers and in 10 doctoral dissertations he had supervised. The committee listed what he had done:

Stapel said he was frustrated by messy data that rarely led to clear conclusions, and wanted results that looked elegant. That is worth pausing on, because the pull he describes is one every researcher feels: real data is untidy, and tidy results are rewarded.

The investigation also noted that statistical flaws in his work revealed a lack of familiarity with elementary statistics, and that many co-authors should have spotted the irregularities but trusted him instead. Knowing the basics is a defence, not just a qualification.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 36-37 — the Stapel investigation

51. Quieter fraud: stopping when the data look good

Concept

Some misconduct is much harder to spot than invented data. The book mentions researchers who simply stop collecting data once they have just enough to show what they hoped. Every number they report is real; the stopping rule is what is dishonest.

Figure (svg): The running share of improved subjects for one simulated researcher testing a treatment that does nothing, from subject 1 to subject 100. The share starts high, is exactly 60 percent at subject 20, where a dot marks the point the researcher could stop, and then drifts down to 50 percent at subject 100.

Early on, the share lurches about. Checking after every ten subjects and stopping on a good look catches one of those lurches and reports it as a finding.

Here is one simulated researcher testing a treatment that does nothing: each subject improves with the same chance as a coin landing heads. The researcher checks after every ten subjects and will stop the moment the share improved reaches 60 percent.

After 20 subjects, 12 have improved, exactly 60 percent, and the researcher stops and reports a success. Had the study run to its full 100 subjects, the share would have come back to 50 of 100, exactly what a useless treatment gives.

Early shares jump about because each new subject moves a small total a long way. A researcher who is allowed to choose the moment of stopping is choosing among those jumps, and will naturally choose a flattering one. The honest remedy is to fix the number of subjects before the study begins.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — stopping data collection once the result is favourable

52. How often does peeking manufacture a result?

Worked example

One lucky path proves little. Simulate 10,000 researchers, each testing a treatment that does nothing, and count how many can claim success under two rules. Success means at least 60 percent of subjects improved.

Rule 1: fix the sample at 100, look once

Why: count the simulated researchers reaching 60 of 100.

\[ \dfrac{299}{10\,000} = 0.030 = 3.0\% \]

Rule 2: look after every 10 subjects, stop on success

Why: count those who reach 60 percent at any look.

\[ \dfrac{4\,817}{10\,000} = 0.482 = 48.2\% \]

Compare the two rates

Why: how many times more often does peeking produce a false success?

\[ \dfrac{4\,817}{299} = 16.1 \]

With the treatment doing nothing at all, the fixed rule is fooled about 3 times in a hundred. The peeking rule is fooled nearly half the time, about sixteen times as often, and every one of those false successes would be reported with real data.

Verify the simulation with an exact calculation

Why: the chance of 60 or more heads in 100 fair tosses, and the chance of hitting 60 percent at any of the ten looks, can be computed exactly instead of simulated.

\[ \text{exact: } 0.0284 \text{ and } 0.4805 \quad \text{simulated: } 0.030 \text{ and } 0.482 \]

The exact values sit within a fraction of a point of the simulated ones, so the sixteen-fold inflation is a property of the peeking rule, not of an unlucky simulation.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — stopping once there is just enough

53. Protecting the people who are studied

Concept

When a study uses human participants, ethics and the law both require researchers to look after them. In the United States the Department of Health and Human Services oversees the federal regulations, and every university or research institution must set up an oversight committee called an Institutional Review Board, or IRB.

Figure (svg): A flow diagram: study plan, then IRB review weighing risks against benefits, then written informed consent from each participant, then data collection, then data guarded for privacy. A dashed arrow runs back from data collection to the IRB, labelled unexpected risks are reported.

Nothing is collected until the board approves, and approval is not a one-off: a study that turns out riskier than planned goes back for review.

Every planned study must be approved in advance by the IRB. The law mandates three key protections, and the picture shows where each one bites:

  1. Risks are minimized and reasonable compared with the projected benefits
  2. Informed consent: the risks are clearly explained, subjects consent in writing, and researchers keep the documentation
  3. Privacy: data collected from individuals is carefully guarded

These sound simple and turn out to be hard. Is deleting names enough to protect privacy, when age, postcode and job together might identify someone? What happens when a study does not go as planned and a risk appears that nobody foresaw? The dashed arrow in the picture is the answer to the second question: the study goes back to the board.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — Institutional Review Boards and the three protections

54. Example 1.22: a researcher's afternoon in the community

Worked example

A researcher is collecting data in a community. For each of three things they do, describe the unethical behaviour, how it could affect the reliability of the data, and how it should be corrected.

Figure (svg): A street map of four blocks of houses. One block, where the researcher knows the residents, is outlined in yellow as the only block visited. On it, four houses where no one answered are drawn hollow, and four houses skipped and then filled in with copied answers are marked with red crosses.

Three failures, one map: the block was chosen for comfort, the empty houses were dropped, and the skipped houses were filled with other people's answers.

(a) Chooses a block where they know many residents

Why: a convenience sample, chosen for comfort, and then presented as the community.

\[ \text{convenience sample} \Rightarrow \text{possibly biased} \]

The people on a familiar street are likely to resemble the researcher and each other, in income, age or background. Claiming they represent the community is misleading. The fix is to choose the areas to visit at random.

(b) Nobody home at four houses; not recorded, not revisited

Why: intentionally leaving out part of the sample.

\[ \text{omitted houses} \Rightarrow \text{missing a whole kind of household} \]

Houses empty in the afternoon are often homes where the adults work. If the survey is about jobs and child care, those are exactly the households it most needs. The fix is to record the addresses and return until the residents are reached.

(c) Skips four houses and fills in answers copied from neighbours

Why: fabricated data, even though each answer was once given by someone real.

\[ \text{duplicated answers} \Rightarrow \text{fraud, and more bias} \]

Check each case against the three failures

Why: (a) biased the choice of sample, (b) biased who was left in it, and (c) invented data; the fixes are random selection, follow-up, and actually interviewing everyone on the route.

\[ \text{(a) selection} \quad \text{(b) omission} \quad \text{(c) fabrication} \quad \checkmark \]

Notice how the three shade into one another. Only one of them is outright fraud, yet all three make the data misrepresent the community, and all three could look perfectly clean in the final spreadsheet.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 38 — Example 1.22

55. Try It 1.22: the teen fruit-juice study

Worked example

A study is commissioned to find the favourite brand of fruit juice among teens in California. For each feature below, describe any unethical behaviour, how it could affect the data, and how to correct it.

(a) Commissioned by the seller of a popular apple juice

Why: a conflict of interest: the sponsor wants a particular answer.

\[ \text{sponsor's interest} \Rightarrow \text{disclose it; keep design independent} \]

(b) Only apple and cranberry juice are included

Why: a question about a favourite brand that offers two juice types cannot find the favourite.

\[ \text{2 options} \Rightarrow \text{answer forced; include the real range} \]

(c) Participants see the brand as samples are poured

Why: brand loyalty and advertising can decide the taste test.

\[ \text{brand visible} \Rightarrow \text{blind the tasting} \]

(d) The results: add the categories the advert combines

Why: 25 percent prefer Brand X and 42 percent have no preference.

\[ 25\% + 42\% = 67\% \text{ like X at least as much} \]

Figure (svg): Left: three bars, prefer Brand X 25 percent, prefer Brand Y 33 percent, no preference 42 percent. Right: two stacked bars. Brand X as much as or more than Y stacks 25 and 42 to make 67; Brand Y as much as or more than X stacks 33 and 42 to make 75.

The same 42 points of indifference can be stacked on either brand. Stacked on Brand Y, the claim is even stronger, which is exactly why Brand X's version misleads.

Combine the same way for Brand Y

Why: 33 percent prefer Y, plus the same 42 percent with no preference.

\[ 33\% + 42\% = 75\% \text{ like Y at least as much} \]

So the commercial's sentence is technically true, and the identical argument gives Brand Y a bigger number. Only 25 percent actually prefer Brand X, the smallest of the three groups. Borrowing the undecided to build a majority misrepresents the result.

Check that the three groups account for everyone

Why: the percentages must add to 100, or the categories overlap or miss someone.

\[ 25 + 33 + 42 = 100 \quad \checkmark \]

They do, so the 42 percent really are the same teens in both stacked bars. That shared middle group is the whole trick, and the honest report gives all three numbers separately.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 38 — Try It 1.22

56. Put an ethical study in order

Ranking

Six stages of a study with human participants, shuffled. Put them in the order a properly run study goes through them, using the IRB picture and the rest of this part.

Put in order

Arrange the stages from first to last.

  1. Write the study plan, including the sample size, before any data exist
  2. Get the plan approved by the Institutional Review Board
  3. Explain the risks to each participant and obtain written consent
  4. Randomly assign participants to treatments
  5. Collect the data, stopping only at the planned sample size
  6. Report every result, including the disappointing ones

Why: The plan comes first, and it fixes the sample size so that nobody can stop at a lucky moment. The board approves it before anyone is recruited. Each participant gives informed consent before being assigned, since they must agree to either treatment. Only then are treatments assigned and data collected, and the study ends by reporting everything, because leaving out unwelcome results distorts the record as surely as inventing data.

The step people most often place wrongly is consent. It has to come before random assignment, because a participant must agree to receive whichever treatment the coin gives them, placebo included.

The first and last steps are the two defences against the quiet fraud of this part. Fixing the sample size in advance blocks the peeking researcher, and reporting everything blocks the researcher who runs five studies and publishes the one that worked.

57. Whose blood is it once the lab is done?

Socratic

The book ends its discussion of ethics with questions, not answers, and one of them is sharper than it looks. Your doctor takes a blood sample to check your cholesterol. Once it is tested, you expect the rest to be thrown away; at that point it is biological waste.

Discussion prompt

Does a researcher have the right to take that leftover blood and use it in a study without asking you? Argue one side, then give the strongest point for the other side, and say which of the three legal protections is at stake.

Notice that the difficulty is not whether consent matters but when it is needed, which is exactly the question the book poses. Real review boards spend much of their time on cases like this.

The book also points to a website, Retraction Watch, that catalogues research articles withdrawn after being shown to be fraudulent. A quick look shows the problem is larger than most people think, and the defence it recommends is knowledge: learning enough statistics to read a study critically.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, p. 37 — questions of consent and privacy

58. Putting it together

Section

Part 7

59. The questions that judge any study

Pattern

Everything in this lesson collapses into one decision and three devices. Follow the flow from the left; the first question decides what kind of conclusion is even available.

Figure (svg): A decision flow. Start: can the researcher assign the explanatory variable? If no, an observational study, which shows association only. If yes, randomly assign units to treatments, then add a control group with a placebo, then blind the subjects and the researchers, arriving at a randomized, controlled, double-blind experiment that can support cause.

Each box on the top row answers one threat, named underneath it. The bottom route is honest research too; it just cannot end with the word causes.
  1. Name the parts: population, sample, units, explanatory variable, treatments, response
  2. Who assigned the explanatory variable? If no one did, it is observational: report an association and name a lurking variable
  3. If it was assigned, was it random? Random assignment answers lurking variables
  4. Is there a control group with a placebo? It answers the power of suggestion
  5. Who was blinded? Subjects against suggestion, researchers against their own hopes
  6. Was it conducted honestly? Planned sample size, consent, privacy, everything reported

When you write a conclusion, carry the reason with it. Because participants were randomly assigned, the difference can be attributed to the treatment. Because people chose for themselves, the difference may be due to a lurking variable such as exercise. The reason is the answer; the verdict alone earns little.

OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics §1.4, pp. 34-38 — the whole section

60. Check: name the treatments

Check

A quick test of Part 1's vocabulary on a new study. Read carefully: the question asks for one specific part, and three of the options are real parts of the study, just not the one asked for.

\[ \text{120 tomato plants; 40 each get fertilizer A, B, or none; fruit weight measured} \]

Check your understanding

A gardener randomly assigns 120 tomato plants, 40 each, to fertilizer A, fertilizer B, or no fertilizer, and weighs the fruit each plant produces. What are the treatments?

  • A. fertilizer A, fertilizer B, and no fertilizer (correct)
  • B. fertilizer A and fertilizer B only
  • C. the 120 tomato plants
  • D. the weight of fruit produced

Answer: A

Why: The treatments are the values of the explanatory variable, which here is the fertilizer given. It takes three values, and applying no fertilizer is one of them, because those 40 plants are a group the gardener deliberately set up for comparison.

Why B tempts people
This leaves out the no-fertilizer group, the same mistake as calling a control group untreated. Those 40 plants receive a deliberate value of the explanatory variable, so it is a treatment.
Why C tempts people
The plants are the experimental units, the things each measurement is taken on. The treatments are what is applied to them.
Why D tempts people
Fruit weight is the response variable, recorded at the end. Treatments are decided before the study, the response after.

If you left out the no-fertilizer group, go back to the trap in Part 3. The comparison group is part of the design, and naming it as a treatment keeps you honest about what is being compared.

Ask which of the four answers could be decided before any tomato grows. Only the list of fertilizers can, which is the quickest way to recognise treatments in any study.

61. Check: what the design allows you to conclude

Check

This one is about reasoning, not vocabulary. Find the feature of the design that decides the answer before looking at the options.

Check your understanding

A university records that students who use its study centre earn higher grades than students who do not. Which conclusion does this study support?

  • A. Use of the study centre is associated with higher grades, but it may not cause them (correct)
  • B. The study centre causes higher grades, since the difference is large
  • C. The study centre causes higher grades, if the university has thousands of students
  • D. The study centre has no effect on grades

Answer: A

Why: Students chose for themselves whether to use the centre, so nobody assigned the explanatory variable and this is an observational study. Motivated students may be both more likely to use the centre and more likely to earn good grades, so the most the data support is an association.

Why B tempts people
The size of a gap says nothing about its cause. A lurking variable such as motivation can produce a large gap as easily as a small one.
Why C tempts people
More students make the association more precise, but the same motivated students still choose the centre. Sample size cannot turn self-selection into random assignment.
Why D tempts people
The study does not show the centre is useless either. It cannot separate the centre's effect from the students' motivation in either direction.

The deciding feature was a single phrase: students who use the centre. Whenever the groups are defined by what people chose to do, you are looking at an observational study.

The option about thousands of students is the one to learn from, because it tempts people who understood Part 2 only halfway. Size buys precision, and precision about a biased gap is still a biased gap.

62. Check: which device answers the threat

Check

A design question. Identify the specific threat described, then pick the device aimed at that threat, not simply the device that sounds most rigorous.

Check your understanding

In a trial of a new cough syrup, patients are randomly assigned and given identical bottles, but the doctor who scores each patient's cough knows which syrup each patient received. Which change would fix the remaining problem?

  • A. Keep the scoring doctor unaware of which syrup each patient received (correct)
  • B. Recruit twice as many patients
  • C. Let patients choose which syrup they take
  • D. Replace the placebo syrup with no syrup at all

Answer: A

Why: Random assignment and identical bottles already handle lurking variables and suggestion. The remaining threat is the doctor's expectation shaping the cough scores, and the device aimed at that is blinding the researcher, which makes the trial double-blind.

Why B tempts people
More patients reduce chance variation, but the doctor's knowledge would bias every score in the same direction, so a bigger trial would just measure the bias more precisely.
Why C tempts people
Letting patients choose undoes the random assignment and brings lurking variables back, which makes things worse, not better.
Why D tempts people
Removing the placebo would unbalance the experience of treatment between the groups, adding the power of suggestion as a second problem.

Match threat to device every time. The doctor knowing is a knowledge problem, and only a knowledge fix, blinding, touches it.

Notice that every wrong option either does nothing to the real threat or reintroduces a threat that was already solved. A tempting fix that damages the design elsewhere is one of the commonest mistakes in study design.

63. Say why random assignment earns the word causes

Explain it to yourself

One argument sits at the centre of this lesson. If you can say it in your own words, you can rebuild the rest of it whenever you need to.

Discussion prompt

In three or four sentences, explain why a randomized experiment can support a claim of cause when an observational study of the same question cannot, even a much larger one. Use a lurking variable in your explanation.

Check your own version for two things. It should say who chooses the treatment in each kind of study, and it should explain why sample size does not rescue the observational one. Those are the two points an examiner looks for.

If your explanation said the groups become identical, soften it. They become alike apart from chance, and chance differences are small and not tilted towards either treatment, which is all the argument needs.

64. Exit ticket: design the sleep study

Exit ticket

One design, from the question to the conclusion you would be entitled to write. Use every device from this lesson, and name the threat each one answers.

Discussion prompt

Design a study of whether a magnesium supplement improves sleep in adults. Name the explanatory variable, the treatments and the response; say how you would assign them, what the control group gets, and who is blinded. Then write the conclusion you could draw if the supplement group improved by 12 points more than the control group.

The conclusion is the part to compare most carefully with the model. It names the design features that entitle it to the word attributed, and it states the effect in percentage points, in context.

If your design let volunteers choose their tablets, or gave the control group nothing, go back to the matching part of the lesson. Each of those choices reopens a threat that the design was supposed to close.

65. What you can do now

Recap

This lesson turned the vague question of whether a study can be trusted into a short list of specific checks, each tied to a specific threat.

the questionwhat answers itwhat it earns
what are the parts of the study?population, sample, units, explanatory variable, treatments, responsea clear statement of what is compared
could something else explain the difference?random assignment spreads lurking variablesthe right to say causes
could expectation explain it?a control group given a placebothe treatment's own effect, by subtraction
could anyone's knowledge leak in?blinding: single for subjects, double for researchers tooa comparison free of hopes
can the variable be assigned at all?if not, an observational studyassociation only
was it conducted honestly?IRB approval, consent, privacy, a fixed plandata worth analysing

\[ \text{random assignment} \Rightarrow \text{cause} \qquad \text{random selection} \Rightarrow \text{generalisation} \]

That line is the lesson's spine: two random steps, two different jobs. A study can earn one without the other, and a good report says which it earned.

The simulations in this deck, the coin splits, the mound of chance gaps and the peeking researchers, are previews. Later chapters measure how wide that mound is and use it to decide when a difference between treatment groups is too large to be chance, which is exactly the question every experiment finally asks.

OpenStax Introductory Business Statistics 2e, §1.4 Experimental Design and Ethics §1.4 — the same design ideas applied to business studies

Sources

  1. OpenStax Introductory Statistics 2e, §1.4 Experimental Design and Ethics — Illowsky & Dean, OpenStax / Rice University, CC BY 4.0, pp. 33-38
  2. OpenStax Introductory Business Statistics 2e, §1.4 Experimental Design and Ethics — Illowsky & Dean, OpenStax / Rice University, CC BY 4.0

Want this taught 1-on-1? Alexander tutors Statistics — $55/session, free consultation.

Book on Wyzant · Text (657) 465-8108