The section that supplies the logic behind a hypothesis test. The reasoning is the rare-event argument: make an assumption about the population, gather data, and if the sample has properties that would be very unlikely to occur when the assumption is true, conclude that the assumption is probably incorrect. The p-value measures that unlikeliness — it is the probability that, if the null hypothesis is true, another randomly selected sample would give results as extreme or more extreme than the ones obtained. A large p-value indicates the null should not be rejected, and the smaller the p-value the stronger the evidence against it. Comparing the p-value against a significance level chosen before the data were collected turns the measurement into a decision, and the conclusion must then be written in the words of the problem. The p-value is conditional on the null being true, which is what makes almost every common misreading of it wrong.
Subject: Statistics · 67 slides · symbolic lesson
Open the interactive version of this deck
Title
Statistics · Chapter 9 — Hypothesis Testing with One Sample
How surprising is the sample, if the null were true, and what may you say about it?
Objectives
Sections 9.1 to 9.3 set up the pieces of a test: a pair of hypotheses, the two errors you might make, and the distribution to use. None of that yet tells you how to decide. This section supplies the reasoning that turns a sample into a decision, and the sentence that reports it.
The arithmetic in this section is the arithmetic of Chapter 7: a standard error, a z-score, an area under a normal curve. What is new is the logic around it. Most marks lost on hypothesis tests are lost in the logic, not the arithmetic, so the logic gets most of the slides.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-469 — the section this deck covers
Warm-up
Chapter 7 gave you the sampling distribution of a mean. Use it on a question that will turn out to be this whole section in disguise.
Discussion prompt
A population has mean 50 and standard deviation 10. You take a random sample of 25. What is the probability that the sample mean is 54 or more?
Now turn the question round. Suppose someone claims the population mean is 50, and your sample of 25 comes back with a mean of 54. You have just computed how surprising that sample would be if the claim were true. That probability, 0.0228, is a p-value, and the rest of this lesson is about what you may and may not conclude from it.
OpenStax Introductory Business Statistics 2e, §9.4 Full Hypothesis Test Examples §7.2, the Central Limit Theorem for sample means — the standard error used here
Section
Part 1
Concept
Every hypothesis test in this book runs on one piece of reasoning. You make an assumption about the population: that assumption is the null hypothesis. Then you gather a random sample.
If the sample has properties that would be very unlikely to occur if the assumption were true, you conclude that the assumption is probably incorrect.
Figure (svg): A flow diagram of three boxes joined by arrows. Box 1, assume: the null hypothesis is true. Box 2, compute: how likely is a sample at least this extreme, if the assumption holds? Box 3, judge, which splits into two exits: a very small probability leads to a red box, doubt the assumption, reject H0; a probability that is not small leads to a grey box, no verdict, do not reject H0.
Look at the two exits from the last box. Only the left one says anything. A very unlikely sample gives you grounds to doubt the assumption. A sample that would be quite ordinary under the assumption gives you no verdict at all, because ordinary samples are also what many other assumptions would produce.
rare-event argument — If a sample would be very unlikely when an assumption about the population is true, the assumption is probably incorrect. The probability is always computed as if the assumption held.
Keep the book's parenthesis in mind as you go. Your assumption is just an assumption; it may or may not be true. Your sample data are real, and they are showing you a fact that seems to contradict the assumption. When a guess and a fact disagree, the guess gives way.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Rare Events
Worked example
At a wealthy friend's birthday party, Didi and Ali are told there are 200 plastic bubbles in a tall basket, and only one holds a hundred-dollar bill. Blindfolded, Didi reaches in first and pulls out a bubble. It contains the bill. What should Ali make of that?
Run the argument in its three moves. First write down the assumption you were given, then work out how likely Didi's draw is if that assumption is true, then judge the number.
State the assumption
Why: what they were told plays the part of the null hypothesis.
\[ \text{assumption: exactly one bill among } 200 \text{ bubbles} \]
Count the ways the observed event can happen
Why: one bubble out of 200 equally likely ones holds the bill.
\[ P(\text{first bubble holds the bill}) = \frac{1}{200} \]
Write it as a decimal
Why: so it can be judged against other probabilities.
\[ \frac{1}{200} = 0.005 \]
Judge it
Why: five chances in a thousand is a rare event.
\[ 0.005 \text{ is very small} \]
So Ali doubts what they were told and hopes there are more bills in the basket. Notice what he doubts: not the draw, which he watched happen, but the story about the basket, which nobody has checked. That is the rare-event argument exactly.
Notice that the probability was computed entirely inside the story they were told. Nobody needs to know how many bills are really in the basket to get 0.005; you only need the assumption. That is what makes the calculation possible, and it is why the answer is a verdict on the assumption.
Check the probability by counting parties
Why: over 200 such parties with one bill in the basket, the first draw should win about once, and lose the rest of the time.
\[ 200 \times 0.005 = 1, \qquad 200 \times 0.995 = 199 \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Didi and Ali
Picture it
Ali's doubt has a direction: he suspects more bills, not fewer. The chart shows why. Each bar is the probability that the first bubble holds a bill, for a different number of bills in the basket of 200.
Figure (svg): A bar chart. Horizontal axis: the number of hundred-dollar bills in a basket of 200 bubbles, at 1, 2, 5, 10, 20 and 50. Vertical axis: the probability that the first bubble drawn holds a bill, k over 200. The bar for 1 bill is red and tiny, 0.005. The bars rise through 0.01, 0.025, 0.05 and 0.10 to 0.25 for 50 bills.
Under the story they were told, one bill, Didi's draw has probability 0.005: the red bar is barely visible. With ten bills it would be 0.05, with fifty it would be 0.25, a one-in-four event that would surprise nobody.
\[ P(\text{first bubble holds a bill}) = \frac{k}{200} \]
This is the comparison hiding inside every test. The observed fact fits the stated assumption badly and fits other assumptions much better. The test never proves which other assumption is right; it only says that the stated one has been made to look doubtful.
It also shows the limits. Even with fifty bills in the basket, three draws in four would come up empty. So a single lucky draw cannot tell Ali how many bills there are, only that one bill is a poor explanation of what he saw.
Intuition
Why should the assumption give way, rather than the data? Because they have different standing. The null hypothesis is something you wrote down before looking. The sample is something that actually happened.
\[ \text{unlikely under } H_0 \;+\; \text{the data are real} \;\Longrightarrow\; \text{doubt } H_0 \]
Think of a friend who swears a coin is fair, and then it lands heads ten times running. You do not decide that the ten heads did not happen. You decide that the friend's claim about the coin is in trouble.
The same asymmetry runs the other way. If the coin had landed heads six times out of ten, you would not decide the coin is fair. Six heads is what a fair coin produces, and also what a slightly biased coin produces, so it settles nothing.
That is why the decisions in Section 9.1 were worded the way they were: reject the null, or do not reject it. There is no third option called accept, because ordinary data cannot single out the null from its neighbours.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — the parenthesis on assumptions and data
Prediction
A friend hands you a coin and says it is fair. You flip it ten times and get ten heads. Commit to an answer before you work anything out.
Ask yourself which of the two things you have is a fact and which is a claim, then decide which should give way.
Predict first
What is the most reasonable conclusion?
Correct: The claim that the coin is fair is now doubtful
Why: Ten heads from a fair coin has probability one in 1,024. That is rare enough to cast serious doubt on the claim of fairness, but rare is not impossible, so the honest word is doubtful rather than certainly biased.
Two of the wrong answers are the two ways people mishandle evidence. One refuses to let any data move the assumption. The other treats a rare event as an impossible one and leaps to certainty.
The rare-event argument sits between them: it lets the data count against the claim, in proportion to how unlikely the data would be if the claim were true. The next slide puts a number on it.
Worked example
A coin is claimed to be fair. Ten flips give ten heads. Find the probability of this result if the claim is true, and decide whether the claim is in doubt.
The flips are independent, so the probability of a run of heads is a product of ten one-halves. Write the product before evaluating it, so you can see where the number comes from.
State the assumption
Why: fair means each flip lands heads with probability one half.
\[ H_0: p = 0.5 \]
Multiply across independent flips
Why: ten heads in a row means heads on every one.
\[ P(\text{10 heads}) = \left(\tfrac{1}{2}\right)^{10} \]
Evaluate the power
Why: two to the tenth is 1,024.
\[ \left(\tfrac{1}{2}\right)^{10} = \frac{1}{1024} \]
Write it as a decimal
Why: about one in a thousand.
\[ \frac{1}{1024} \approx 0.000977 \]
In context: if the coin really were fair, ten heads in ten flips would happen about once in every thousand sets of ten. That is rarer than Didi's draw, so the claim of fairness is in more serious doubt than the story about the basket was.
Compare the rare events met so far. Didi's draw was 1 in 200; four sixes in a row on a fair die would be about 1 in 1,300; ten heads is about 1 in 1,000. None is impossible, and each is rare enough that most people would stop trusting the claim behind it.
Check against the binomial distribution
Why: the binomial formula with n equal to 10 and x equal to 10 must give the same number.
\[ \binom{10}{10}(0.5)^{10}(0.5)^{0} = 1 \times \frac{1}{1024} \times 1 = 0.000977 \]
Counterexample
Here is a puzzle that breaks a careless version of the argument. Flip a fair coin ten times and write down the exact sequence. Suppose it comes out heads, tails, heads, heads, tails, tails, heads, tails, heads, tails.
\[ P(\text{HTHHTTHTHT}) = \left(\tfrac{1}{2}\right)^{10} = 0.000977 \]
That sequence is exactly as unlikely as ten heads. Yet nobody would doubt the coin after seeing it.
Discussion prompt
If every exact sequence of ten flips has probability 0.000977, why does ten heads cast doubt on the coin while this mixed sequence does not?
Figure (svg): A probability histogram for the number of heads in 10 flips of a fair coin, bars at 0 to 10, symmetric and peaking at 5 with height 0.246. The bars at 9 and 10 heads are red; the bar at 10 is labelled 0.001 and the pair is labelled 9 or more: 0.0107.
This is why the p-value, coming up in Part 2, is defined with the words as extreme or more extreme. A single exact outcome is almost always improbable. A tail of outcomes, all pointing away from the null in the same direction, is what measures surprise.
Trap
After Didi's draw, a first attempt at Ali's conclusion says:
\[ P = 0.005 \]
\[ \Rightarrow \text{ there must be more bills} \]
The probability is right. The word must is the error: it treats a one-in-two-hundred event as if it could not happen.
Too strong. Rare events do happen.
With one bill in the basket, the first draw still wins once in 200 parties. Didi may simply have been lucky. What the argument gives is grounds for doubt, not proof.
\[ P = 0.005 \]
\[ \Rightarrow \text{ the one-bill story is doubtful} \]
That is also why every test carries a chance of a Type I error from Section 9.2: rejecting an assumption that was true all along, exactly when the rare event happens.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Didi and Ali
Sorting
For each situation, compute or estimate how likely the observation would be if the stated assumption were true. Then decide whether it casts doubt on the assumption or gives no verdict.
Sort into buckets
Sort each observation by what it does to the stated assumption.
Look at the three in the no-verdict bucket. None of them confirms its assumption; a slightly unfair coin or a machine averaging 7.95 ounces would produce the same data just as easily. Ordinary data are consistent with the assumption and with its neighbours.
Notice too that the two means needed a standard error before they could be judged. A mean of 7.98 ounces sounds low until you see it is only a quarter of a standard error below 8. From here on, every judgement of rarity goes through the sampling distribution.
Section
Part 2
Concept
Part 1 judged rarity by eye. A test needs a number, and the book names it. Use the sample data to calculate the probability of getting the test result, called the p-value.
p-value — The probability that, if the null hypothesis is true, the results from another randomly selected sample will be as extreme or more extreme as the results obtained from the given sample.
\[ p\text{-value} = P(\text{a result at least as extreme as yours} \mid H_0 \text{ true}) \]
Read the vertical bar as given that. The whole calculation lives in an imaginary world where the null hypothesis is true. You build that world's sampling distribution, place your sample in it, and measure the area beyond.
Then read the size of the number. A large p-value means your sample is the kind of thing the null produces routinely, so you should not reject the null. The smaller the p-value, the more unlikely the outcome, and the stronger the evidence against the null.
The book adds a piece of advice you should take literally: draw a graph that shows the p-value. The test is easier to perform with a picture, because you see the problem more clearly, and every worked example in this deck has one.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Using the Sample to Test the Null Hypothesis
Notation
Here is the p-value for a right-tailed test of a mean, the kind in Example 9.9. Every symbol in it carries part of the definition, so decode each one before using it.
Annotate
On: \( p\text{-value} = P(\bar{X} \ge \bar{x}_{\text{obs}} \mid \mu = \mu_0) \)
The capital and small letters matter here. The capital X-bar is the random quantity that varies from sample to sample; the small x-bar with its label is the one value you actually got. The p-value asks how often the random one would land beyond the fixed one.
And the condition after the bar is not decoration. Take it away and the sentence stops being something you can compute, because without assuming a value for the mean there is no curve to find an area under.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-468 — the definition and Example 9.9
Concept
You might ask why the p-value includes results more extreme than yours. Why not just the probability of the sample you got?
The first reason is the coin from Part 1: an exact outcome is nearly always improbable, so its probability cannot measure surprise. The second is sharper. For a continuous variable such as a sample mean, the probability of any one exact value is zero.
\[ P(\bar{X} = 17) = 0 \quad \text{for every possible population mean} \]
A probability that is zero whatever the truth is cannot count against anything. So the book's definition takes the area of the whole tail beyond your result: all the samples that would be at least as surprising as yours.
\[ P(\bar{X} \ge 17 \mid \mu = 15) \]
Read that tail as a ranking. Your sample sits at some point on the null's curve. The p-value is the fraction of all possible samples that would rank as at least as extreme. A p-value of 0.01 says only one sample in a hundred would rank as far out as yours.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the p-value as an area beyond the sample mean
Worked example
A baker claims that his bread height is more than 15 cm, on average. Several customers do not believe him. He bakes 10 loaves; their mean height is 17 cm. From baking hundreds of loaves he knows the standard deviation of height is 0.5 cm and the heights are normal. Set up the test and the null's sampling distribution.
This slide does the setting up; the next one does the probability. The order matters, because the curve you draw depends on the hypotheses you wrote, not on the sample.
Write the alternative from the claim
Why: is more than translates as greater than, and the claim the baker wants to show goes in the alternative.
\[ H_a: \mu > 15 \]
Write the null as its complement
Why: the null must contradict the alternative and carry the equality.
\[ H_0: \mu \le 15 \]
Choose the distribution
Why: sigma is known and heights are normal, so the sample mean is exactly normal: a z-test, Section 9.3's first row.
\[ \bar{X} \sim N\left(\mu, \tfrac{\sigma}{\sqrt{n}}\right) \]
Compute the standard error
Why: sigma over the square root of the sample size.
\[ \frac{\sigma}{\sqrt{n}} = \frac{0.5}{\sqrt{10}} = \frac{0.5}{3.1623} = 0.1581 \]
Centre the curve at the null's boundary value
Why: assume the null is true at its edge, a mean of 15.
\[ \bar{X} \sim N(15,\; 0.1581) \]
In context: if the baker's loaves really average 15 cm, the means of samples of 10 loaves pile up around 15 with a standard error of about 0.16 cm. That is the imaginary world in which the next slide asks how surprising 17 cm would be.
Notice that the sample mean, 17, has not been used yet. The null curve is built from three things only: the null value, sigma and the sample size. If you ever centre the curve at your own sample mean, every p-value you get from it will be about one half, and the test will have asked nothing.
Check the standard error by undoing the division
Why: multiplying back by the square root of 10 must return sigma.
\[ 0.1581 \times 3.1623 = 0.5000 = \sigma \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-468 — Example 9.9
Concept
The baker's null says the mean is at most 15, a whole range of values. Yet the curve was centred at exactly 15. Why that one value, and not 14.8 or 14?
Because the boundary is the null's best case. Any mean below 15 would slide the curve further left and make the observed sample even more surprising. Try It 9.9, coming up shortly, shows it with numbers you can see.
\[ \mu = 12: \quad z = \frac{12.5 - 12}{1/6} = 3, \quad p = 0.00135 \]
\[ \mu = 11.9: \quad z = \frac{12.5 - 11.9}{1/6} = 3.6, \quad p = 0.00016 \]
Moving the assumed mean down by a tenth makes the p-value about eight times smaller. So the p-value computed at the boundary is the largest one any value in the null could give. If even that is small, every version of the null is in trouble at once.
That is why the equality sits in the null and the calculation happens at the boundary. It is the most generous reading of the null, which keeps the test fair: the null is rejected only when its strongest version fails.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the distribution of sample means centred at μ = 15
Worked example
With the null curve in hand, N(15, 0.1581), find the p-value for the observed mean of 17 cm and say what it means.
The alternative points right, so more extreme means larger. The p-value is the area of the null's curve to the right of 17. Turn 17 into a z-score first, so the area can be read from the standard normal.
Write the p-value as a tail
Why: the probability another sample mean is the same or greater than 17 when the population mean is 15.
\[ p\text{-value} = P(\bar{X} \ge 17 \mid \mu = 15) \]
Standardise the boundary
Why: the distance from the null value, in standard errors.
\[ z = \frac{17 - 15}{0.1581} = \frac{2}{0.1581} \]
Divide
Why: 12.65 standard errors above the centre.
\[ z = 12.65 \]
Read the right tail
Why: far beyond any table; node puts it near 6 times 10 to the minus 37.
\[ p\text{-value} = P(Z \ge 12.65) \approx 5.7 \times 10^{-37} \approx 0 \]
Figure (svg): A narrow bell curve centred at 15 cm, the sampling distribution of the mean of 10 loaves if the null were true, on an axis from 14.4 to 17.3. The curve rises and falls between about 14.5 and 15.5 and lies flat on the axis beyond that. A red dashed line marks the observed mean 17, far to the right, labelled 12.6 standard errors out.
In context: if the baker's loaves really averaged 15 cm, essentially no sample of 10 loaves would have a mean as high as 17 cm by chance alone. The outcome is so unlikely under the null that the evidence is strongly against it.
A number like 5.7 times 10 to the minus 37 is hard to picture. If every person on Earth had baked a batch of ten loaves every second since the universe began, you would still not expect a single batch averaging 17 cm. The book's approximately zero is exactly right.
Check the size with the three-standard-error band
Why: almost every sample mean lies within 3 standard errors of the centre, and 17 is far outside that band.
\[ 15 + 3(0.1581) = 15.47 \ll 17 \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9 and Figure 9.2
Picture it
Two centimetres above 15 does not sound dramatic. Why is the p-value so absurdly small? The picture puts the two relevant curves on one axis.
Figure (svg): Two bell curves centred at 15 cm on one axis from 13 to 17.6. The wide orange curve, single loaves with standard deviation 0.5, reaches out to about 16.5. The tall narrow blue curve, means of 10 loaves with standard error 0.158, is confined between about 14.5 and 15.5. A red dashed line marks 17 cm.
The wide orange curve is single loaves, standard deviation 0.5. For one loaf, 17 cm is four standard deviations out: rare, about 3 loaves in 100,000, but loaves like that exist.
\[ \text{one loaf: } z = \frac{17 - 15}{0.5} = 4, \qquad \text{mean of ten: } z = \frac{17 - 15}{0.1581} = 12.65 \]
The narrow blue curve is means of ten loaves. Averaging cancels out the high and low loaves, so the means huddle within half a centimetre of 15. For an average of ten to reach 17, every loaf would have to be tall at once.
This is also where the book's wording slips. It says almost 0 percent of all loaves of bread would be at least as high as 17 cm. The calculation is about means of samples of ten loaves, not about single loaves. The precise sentence is: almost no sample of ten loaves would average 17 cm or more if the population mean were 15.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9's interpretation, restated for sample means
Worked example
A p-value of essentially zero makes the logic look trivial. Keep everything in Example 9.9 the same, but suppose the ten loaves had averaged 15.3 cm. Find the p-value.
Only the boundary of the tail moves. The null curve is still N(15, 0.1581), because the curve comes from the null and the sample size, not from the sample's mean.
Write the tail
Why: same direction, new boundary.
\[ p\text{-value} = P(\bar{X} \ge 15.3 \mid \mu = 15) \]
Standardise
Why: the distance is now 0.3 cm.
\[ z = \frac{15.3 - 15}{0.1581} = \frac{0.3}{0.1581} = 1.897 \]
Read the right tail
Why: from the standard normal table.
\[ P(Z \ge 1.897) = 1 - 0.9711 = 0.0289 \]
Figure (svg): The same null curve for means of 10 loaves, centred at 15 with standard error 0.158, on an axis from 14.45 to 15.55. A red dashed line marks a sample mean of 15.3, z equals 1.90, and the visible right tail beyond it is shaded red and labelled p equals 0.0289.
In context: if the loaves really averaged 15 cm, about 3 samples of ten in every 100 would average 15.3 cm or more. That is unusual, but not wildly so, and whether it counts as enough evidence depends on the standard you set before looking, which is Part 3's subject.
Compare this with Example 9.9. The curve did not move, the sample size did not change, and the standard error is the same 0.1581. Only the distance from 15 shrank, from 2 cm to 0.3 cm, and the p-value went from essentially zero to about 3 in 100.
Check with the 68-95-99.7 rule
Why: 1.897 standard errors is just short of 2, where the one-sided tail is 0.025, so the answer should be a little above 0.025.
\[ \tfrac{1}{2}(1 - 0.95) = 0.025 < 0.0289 < \tfrac{1}{2}(1 - 0.68) = 0.16 \]
Worked example
A normal distribution has standard deviation 1. You want to verify a claim that the mean is greater than 12. A sample of 36 has mean 12.5. The hypotheses are given. Find the p-value and draw it.
\[ H_0: \mu \le 12, \qquad H_a: \mu > 12 \]
The same four moves as the baker: the standard error, the z-score, the tail, the picture. This time the answer is small but not invisible, so the picture earns its keep.
Standard error
Why: sigma over the square root of 36.
\[ \frac{\sigma}{\sqrt{n}} = \frac{1}{\sqrt{36}} = \frac{1}{6} = 0.1667 \]
Standardise the sample mean
Why: half a unit above 12, in sixths.
\[ z = \frac{12.5 - 12}{1/6} = 0.5 \times 6 = 3 \]
The right tail beyond z = 3
Why: the alternative points right.
\[ p\text{-value} = P(Z \ge 3) = 1 - 0.99865 = 0.00135 \]
Figure (svg): A bell curve centred at 12 with standard error one sixth, axis from 11.4 to 12.7 with a tick every standard error. A red dashed line marks the sample mean 12.5, three standard errors above the centre. The sliver of area to its right is shaded red and labelled p equals 0.00135.
In context: if the true mean were 12, only about 13 or 14 samples of 36 in every 10,000 would have a mean of 12.5 or more. The book prints 0.0013; the four-decimal value is 0.0013 and the exact tail is 0.00135, which matters on a later slide when the level is 0.001.
Look at the tick marks on the figure: one every sixth of a unit, each a standard error. Counting ticks from 12 to 12.5 gives three, which is the z-score read straight off the picture. Marking the axis in standard errors is the fastest way to sketch any p-value.
Check with the 68-95-99.7 rule
Why: 99.7 percent lies within 3 standard errors, so 0.3 percent lies outside, half of it in each tail.
\[ \tfrac{1}{2}(1 - 0.997) = 0.0015 \approx 0.00135 \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Try It 9.9
Tweak it
Here is Try It 9.9's p-value as a function of the sample mean, for a right-tailed test of a mean of 12 with sigma equal to 1. The curve gives the p-value you would get for each possible x-bar. Change n to see what a bigger sample does.
Parameter explorer
With n at 36, read the p-value at sample means of 12.2, 12.3 and 12.5. Then drop n to 9 and read the p-value at 12.5 again. What does a smaller sample do to the evidence from the same sample mean?
\[ p\text{-value} = P\left(Z \ge (\bar{x} - 12)\sqrt{n}\right), \quad n = {n} \]
What to notice: at x-bar equal to 12 the curve sits at exactly 0.5 whatever n is. A sample mean right on the null value is the most ordinary result possible, and half of all samples would beat it.
As x-bar moves right the p-value falls, and it falls faster for a big n. With 100 values, 12.5 is five standard errors out and the p-value is about 3 in 10 million. With 9 values the same 12.5 is only 1.5 standard errors out and the p-value is 0.0668. Evidence depends on the sample mean and on how precisely it was measured.
Ranking
All four samples test the same claim as Try It 9.9: a normal population with sigma equal to 1, the null mean at most 12, the alternative greater than 12. They differ in sample mean and in sample size.
Strength of evidence against the null means a smaller p-value. Work out each standard error and z-score before you order them; the sample size changes the answer.
Put in order
Order the samples from the strongest evidence against the null to the weakest.
Why: With 36 values the standard error is one sixth, so 12.5, 12.3 and 12.2 give z of 3, 1.8 and 1.2 and p-values of 0.00135, 0.0359 and 0.1151. With 9 values the standard error is one third, so 12.5 gives z of 1.5 and a p-value of 0.0668, which falls between the 12.3 and 12.2 samples.
\[ n = 9: \quad z = \frac{12.5 - 12}{1/3} = 1.5, \quad p = 0.0668 \]
The small sample with the large mean is the one that catches people. Its mean is the highest, tied with the strongest sample, yet its evidence is weaker than a sample of 36 averaging only 12.3. A small sample has a wide null curve, so even a big departure is not very surprising.
Concept
As extreme or more extreme needs a direction, and the alternative hypothesis supplies it. The baker's alternative said greater than, so the extreme samples were the large ones and the p-value was a right tail.
Figure (svg): Three bell curves side by side, each centred at mu-nought. Left panel, alternative mu less than mu-nought: the left tail beyond the sample mean is shaded red. Middle panel, alternative mu greater than mu-nought: the right tail is shaded. Right panel, alternative mu not equal to mu-nought: both tails are shaded, the mirror image of the observed distance included.
If the alternative says less than, small sample means are the extreme ones, and the p-value is the left tail. If it says not equal to, a surprise on either side counts, so the p-value collects both tails: the one beyond your sample and its mirror image on the other side.
\[ \text{two-tailed: } p\text{-value} = 2 \times P(Z \ge |z|) \]
Section 9.5 practises all three shapes with full examples. For now, fix one habit: read the sign in the alternative before you shade anything. The sample tells you where the boundary of the tail is; only the alternative tells you which side of it to shade.
For the same z-score a two-tailed p-value is twice the one-tailed one. Try It 9.9's z of 3 would give 0.0027 against a not-equal alternative, instead of 0.00135. The data are the same; the question asked of them is broader.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — left-, right- and two-tailed tests
Trap
For the baker, a first attempt computes the chance of the observed mean itself:
\[ p = P(\bar{X} = 17) \]
\[ = 0 \]
It looks precise, but no sample could ever count against the null this way.
Wrong question. It is zero whatever the truth is.
A sample mean is continuous, so any one exact value has probability zero under every hypothesis. The p-value is the area of the tail beyond your value.
\[ p = P(\bar{X} \ge 17 \mid \mu = 15) \]
For Example 9.9 both happen to come out near zero, which hides the mistake. For Try It 9.9 the exact-value version still gives zero, while the tail gives 0.00135, and for a sample mean of 12.2 the tail is 0.1151, nowhere near zero.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the p-value is the same or greater than 17
Estimation
A right-tailed test of a mean gives a sample mean exactly two standard errors above the null value. You have no table. Estimate.
Predict first
Roughly what is the p-value?
Correct: About 0.025
Why: About 95 percent of a normal curve lies within two standard deviations of its centre, leaving 5 percent outside, half of it in each tail. A right-tailed test counts one tail, so the p-value is about 0.025; the exact value is 0.0228.
\[ P(Z \ge 2) = 1 - 0.9772 = 0.0228 \approx \tfrac{1}{2}(1 - 0.95) \]
Keep three anchors from the empirical rule. One standard error out gives a one-tailed p-value near 0.16; two give about 0.025; three give about 0.0015. Any p-value you compute should agree roughly with these, and if it does not, check the direction of the tail and the standard error.
The anchor 0.16 is worth noticing too. A sample mean a whole standard error above the null value sounds like a departure, yet one sample in six would do that by chance alone.
Section
Part 3
Concept
A p-value is a measurement. A decision needs a standard to measure it against. The book's systematic way is to compare the p-value with a preset or preconceived alpha, also called the significance level.
significance level α — The probability of a Type I error, rejecting the null when it is true, chosen before the data are collected. It may or may not be given in the problem; if none is given, 0.05 is a common standard.
\[ \alpha > p\text{-value} \;\Rightarrow\; \text{reject } H_0 \qquad \alpha \le p\text{-value} \;\Rightarrow\; \text{do not reject } H_0 \]
Figure (svg): A number line of p-values from 0 to 0.13. A yellow dashed line at 0.05 marks alpha, and the region to its left is tinted red and labelled reject H0 at 0.05. A pink dotted line marks 0.01. Four dots: 0.00135 and 0.025 and 0.0304 lie left of 0.05 in red; 0.1151 lies right of it in blue, labelled do not reject.
When you reject, the results of the sample data are called significant: there is sufficient evidence to conclude that the null is an incorrect belief and that the alternative may be correct. When you do not reject, the results are not significant: there is not sufficient evidence for the alternative.
Read the equality case carefully. The book puts a p-value exactly equal to alpha on the do-not-reject side. To reject, the p-value must be strictly smaller than alpha.
Look at the figure with both fences. At alpha equal to 0.05, three of the four p-values reject. At alpha equal to 0.01, only the smallest does: the dots at 0.025 and 0.0304 cross to the do-not-reject side. Nothing about the data changed; only the standard did.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Decision and Conclusion
Notation
The rule is short enough to memorise and easy to turn upside down. Decode it piece by piece.
Annotate
On: \( \alpha > p\text{-value} \;\Rightarrow\; \text{reject } H_0 \)
The book's memory device says the same thing in a rhyme: if the p-value is low, the null must go; if the p-value is high, the null must fly. Low and high here mean below or above alpha, not small or large in any absolute sense.
A p-value of 0.03 is low against an alpha of 0.05 and high against an alpha of 0.01. The rhyme only works once you have said which alpha you are comparing with.
One more caution about direction. Memorise the rule as p greater than alpha means reject and you will get every decision backwards. Anchor it to meaning instead: a smaller p-value is stronger evidence, so rejection belongs to the small p-values.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Example 9.10's memory device
Worked example
The book stops Example 9.9 at the p-value, which is essentially zero. No significance level is given. Make the decision and say what it means.
Section 9.5 gives the default to use when no level is stated: alpha equal to 0.05. Write it down before comparing, so the reader knows the standard that was applied.
Fix the level
Why: none is given, so use the common standard and say so.
\[ \alpha = 0.05 \]
Recall the p-value
Why: from the previous calculation.
\[ p\text{-value} \approx 5.7 \times 10^{-37} \]
Compare
Why: alpha is far larger than the p-value.
\[ 0.05 > 5.7 \times 10^{-37} \]
Apply the rule
Why: alpha greater than the p-value means reject.
\[ \text{reject } H_0: \mu \le 15 \]
In context: the results are significant. At the 5 percent level there is sufficient evidence that the population mean height of the baker's loaves is greater than 15 cm. With a p-value this small, the decision would be the same at any level anyone uses.
The remark about any level is worth unpacking. Rejection needs alpha above the p-value, and nobody uses an alpha anywhere near 10 to the minus 37. When a p-value is this extreme the choice of level stops mattering; it matters only for results near the fence.
Check with the cutoff for the sample mean
Why: at the 0.05 level the rejection region starts 1.645 standard errors above 15, and 17 is far beyond it.
\[ 15 + 1.645(0.1581) = 15.26 < 17 \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9's conclusion
Fill the middle
The book's Example 9.10 asks you to complete four sentences. Fill in the missing words, using comparisons with alpha.
Fill in the blanks
Reject the null hypothesis when the p-value is less than alpha. The results of the sample data then support the alternative hypothesis. Do not reject the null hypothesis when the p-value is greater than or equal to alpha.
Why: A p-value below alpha rejects the null and the sample data support the alternative. A p-value at or above alpha does not reject, and the data do not support the alternative. The book's solution writes greater than alpha for the second case; the rule on page 468 adds the equal case to it.
Two small repairs to the book here. Its blank reads do not reject the null when hypothesis when, a typo for do not reject the null hypothesis when. And its solution says greater than alpha, leaving out a p-value exactly equal to alpha; the decision rule on the previous page puts that case on the do-not-reject side, so the sentence above says greater than or equal to.
Notice the careful verb in the second sentence. The data support the alternative; they do not prove it. Support is the strongest word a single test earns.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Example 9.10
Worked example
It's a Boy Genetics Labs claim their procedures improve the chances of a boy being born. A test of a single population proportion gives the results below. Interpret them and state a conclusion in simple, non-technical terms.
\[ H_0: p = 0.50, \quad H_a: p > 0.50, \quad \alpha = 0.01, \quad p\text{-value} = 0.025 \]
Two different p's appear in that line, which is a nuisance of notation. The p in the hypotheses is the population proportion of boys; the p-value is a probability about samples. Keep them apart before comparing anything.
Name the parameter
Why: p is the proportion of births that are boys under the labs' procedures.
\[ p = \text{proportion of boys born} \]
Compare alpha with the p-value
Why: the level was set at 1 percent.
\[ \alpha = 0.01 < 0.025 = p\text{-value} \]
Apply the rule
Why: alpha is at or below the p-value, so do not reject.
\[ \text{do not reject } H_0 \]
Say what the p-value meant
Why: if the procedures did nothing, about 2.5 percent of samples would show this many boys or more.
\[ P(\text{this many boys or more} \mid p = 0.5) = 0.025 \]
In plain terms: at the 1 percent level, the data do not give enough evidence that the labs' procedures raise the chance of having a boy. The results were somewhat unusual for an ordinary fifty-fifty chance, but not unusual enough to meet the strict standard the test set.
The word simple in the question matters. A parent reading this result does not know what a null hypothesis is, so the conclusion has to talk about babies and boys, not about H0 and alpha.
Check what the conclusion does not say
Why: not rejecting is not evidence that the procedures fail; at alpha equal to 0.05 the same p-value would have rejected, which is why the level must be stated.
\[ 0.05 > 0.025 \;\Rightarrow\; \text{reject at the 5 percent level} \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Try It 9.10
Worked example
Try It 9.9's p-value was 0.00135. Decide the test at alpha equal to 0.01 and again at alpha equal to 0.001, and draw both rejection regions.
Same data, two standards. The comparison is one line each; the figure shows why the two answers differ.
At the 1 percent level
Why: compare 0.01 with 0.00135.
\[ \alpha = 0.01 > 0.00135 \;\Rightarrow\; \text{reject } H_0 \]
At the 0.1 percent level
Why: compare 0.001 with 0.00135.
\[ \alpha = 0.001 < 0.00135 \;\Rightarrow\; \text{do not reject } H_0 \]
Find each cutoff for the sample mean
Why: the z for a right tail of 0.01 is 2.326; for 0.001 it is 3.090.
\[ 12 + 2.326 \cdot \tfrac{1}{6} = 12.388, \qquad 12 + 3.090 \cdot \tfrac{1}{6} = 12.515 \]
Figure (svg): The Try It 9.9 null curve, centred at 12 with standard error one sixth. The pink region beyond 12.388 is the rejection region for alpha 0.01; the smaller red region beyond 12.515 is the rejection region for alpha 0.001. A yellow line marks the sample mean 12.5, which falls inside the pink region but just short of the red one.
In context: at the 1 percent level there is sufficient evidence that the mean is greater than 12; at the 0.1 percent level there is not. This is where the book's rounded 0.0013 would mislead you: it is above 0.001 either way, but 0.00135 shows how close the call really is.
The picture shows why both decisions are right at once. Each alpha draws its own rejection region on the null's curve, and the smaller the alpha, the further right the region begins. Comparing the p-value with alpha and comparing the sample mean with the cutoff are the same test seen two ways.
Check the decisions against the cutoffs
Why: the sample mean must lie beyond a cutoff exactly when alpha exceeds the p-value.
\[ 12.5 > 12.388 \;\text{(reject)}, \qquad 12.5 < 12.515 \;\text{(do not reject)} \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Try It 9.9 and the decision rule
Prediction
Two researchers analyse the same data. One tests at alpha equal to 0.05, the other at 0.01. Commit to an answer before reasoning it through.
Predict first
Which researcher is more likely to reject the null hypothesis?
Correct: The one using 0.05
Why: Rejection needs the p-value to fall below alpha. Every p-value below 0.01 is also below 0.05, but p-values between 0.01 and 0.05 reject only for the researcher using 0.05. A smaller alpha is a stricter standard and rejects less often.
People often mix up which way stricter goes. A small alpha means a small chance of a false alarm, so it demands rarer data before rejecting. The price is that real effects are missed more often: the Type II error from Section 9.2 becomes more likely.
So choosing alpha is choosing a balance between the two errors. It has to be done before the data are seen, for a reason a simulation a few slides on makes visible.
Sorting
Apply the decision rule to each pair. Compare alpha with the p-value, and remember which side the equal case falls on.
Sort into buckets
Sort each test result by its decision.
Look at the pair with a p-value of 0.0359. It would reject at 0.05, the default, but this test was set at 0.025, so it does not. The decision belongs to a p-value and a level together; neither number alone can make it.
And the 0.05 against 0.05 pair is the one to remember. It is low by any everyday standard, but the rule asks whether alpha is greater than the p-value, and equal is not greater.
Edge cases
Push the rule to its edge. With continuous data a p-value is almost never exactly equal to alpha, but with rounded numbers, or a discrete count, it can be.
Discussion prompt
A test at alpha equal to 0.05 reports a p-value of exactly 0.0500. What is the decision, and why does the book draw the line on that side?
The tie rule follows the logic of the whole chapter. The null is the default and the alternative must earn its place. A result that only just matches the standard has not exceeded it.
There is a second lesson in the edge case. Results very close to the fence, whichever side they land, carry almost the same evidence. Part 5 comes back to this: the decision is a yes or no, but the evidence is a sliding scale.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — If α ≤ p-value, do not reject H0
Concept
The book insists that the statistician selects alpha before collecting the sample data. The reason is visible in a simulation. Here are 1,000 right-tailed tests in which the null hypothesis is true every time; each gave a p-value.
Figure (svg): A histogram of 1,000 p-values from simulated right-tailed tests in which the null hypothesis is true. Twenty bins each 0.05 wide from 0 to 1, all roughly the same height near 50. The first bin, p-values below 0.05, is red and holds 39. A dashed line marks the expected height 50.
The p-values are spread evenly from 0 to 1. That is a general fact: when the null is true, the p-value is equally likely to fall anywhere. So the fraction falling below 0.05 is about 0.05. In this run 39 of the 1,000 did, close to the 50 expected.
\[ P(p\text{-value} < \alpha \mid H_0 \text{ true}) = \alpha \]
That line is the promise behind alpha: the probability of a Type I error is exactly the level you set. Now break the promise. Suppose you look at the p-value first and then pick a level that makes it significant, using 0.10 whenever the p-value lands between 0.05 and 0.10. In this run 82 true nulls would be rejected, not 39.
Choosing alpha after seeing the data quietly doubles the false-alarm rate in that example, and nothing in the final report reveals it. Fixing alpha first is what makes the significance level mean what it says.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — the statistician selects α before collecting the sample data
Real world
If no level of significance is given, the book says a common standard is alpha equal to 0.05, and you should state it in the conclusion. In practice, the field and the cost of a false alarm set the level.
Particle physicists announce a discovery only at five standard errors, a one-tailed p-value of about 3 in 10 million. Medical trials commonly use 0.05, sometimes 0.01 for a confirmatory trial. A market researcher deciding which of two adverts to run might happily use 0.10.
\[ P(Z \ge 5) \approx 2.9 \times 10^{-7} \]
Discussion prompt
Why would a physicist demand an alpha thousands of times smaller than a market researcher's? Answer in terms of the Type I error from Section 9.2.
Whichever level is chosen, it is chosen in advance and stated in the conclusion. A result reported as significant without its level is incomplete, because significant at 0.10 and significant at 0.001 are very different claims.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — If no level of significance is given, a common standard is α = 0.05
Section
Part 4
Concept
The p-value is computed by assuming the null is true. That one fact rules out the most common misreading of it: the p-value is not the probability that the null hypothesis is true.
\[ p\text{-value} = P(\text{data this extreme} \mid H_0) \;\ne\; P(H_0 \mid \text{data}) \]
Swapping the two sides of a given-that bar changes the question completely. The probability that a person speaks English, given that they are a United States senator, is essentially 1. The probability that a person is a senator, given that they speak English, is tiny. Same two events, very different numbers.
The left-hand quantity is what a test can compute, because the null hands you a curve. The right-hand quantity would need to know how plausible the null was before you collected any data: how many of the claims tested in situations like this are true. The test has no such information.
A null hypothesis is either true or false; it does not have a probability in this framework. What has a probability is the sample, which could have come out differently. The randomness lives in the data, exactly as it did for confidence intervals in Chapter 8.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — the p-value assumes the null hypothesis is true
Worked example
Imagine 1,000 studies, each testing a null at alpha equal to 0.05. Suppose 900 of the nulls are actually true and 100 are false, and each test has power 0.80 against a false null. Among all the rejections, what fraction are true nulls?
Every rejected study reported a p-value below 0.05. If a p-value were the probability that the null is true, at most 5 percent of these rejections could be wrong. Count them and see.
False alarms among the true nulls
Why: each true null is rejected with probability alpha.
\[ 900 \times 0.05 = 45 \]
Real findings among the false nulls
Why: each false null is caught with probability equal to the power.
\[ 100 \times 0.80 = 80 \]
Total rejections
Why: add the two kinds.
\[ 45 + 80 = 125 \]
Fraction of rejections that were true nulls
Why: false alarms over all rejections.
\[ \frac{45}{125} = 0.36 \]
Figure (svg): A tree. The root box: 1,000 nulls tested at alpha 0.05. It branches into 900 true nulls and 100 false nulls. The 900 true nulls branch into 45 rejected, false alarms, in red, and 855 not rejected. The 100 false nulls branch into 80 rejected, real findings, in green, and 20 missed. A bracket gathers the two rejected boxes: 125 rejections, of which 45, that is 36 percent, are true nulls.
In context: 36 percent of the significant results in this field would be false alarms, even though every one passed the 5 percent standard. The figure depends on how many of the tested nulls were true to begin with, which is exactly the information a single p-value does not contain.
Change the field and the answer changes. If half the tested nulls were false instead of a tenth, the false alarms would fall to 25 out of 425 rejections, about 6 percent. The p-values could be identical in both fields; what differs is how plausible the nulls were before anyone looked.
Check that the tree accounts for every study
Why: the four leaves must add back to 1,000.
\[ 45 + 855 + 80 + 20 = 1000 \]
Trap
A report says the p-value was 0.03 and concludes:
\[ P(H_0 \text{ true}) = 0.03 \]
so there is a 97 percent chance the effect is real.
It reads naturally, which is why it turns up so often in news reports.
Wrong. The 0.03 was computed assuming the null is true.
The p-value is a probability about the data, given the null:
\[ P(\text{data this extreme} \mid H_0) = 0.03 \]
Correct reading: if the null were true, a sample this extreme would turn up about 3 times in 100. How likely the null itself is was never computed, and the tree on the previous slide shows it can be far larger than 0.03.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — the definition of the p-value
Error analysis
A test at alpha equal to 0.05 returns a p-value of 0.03. Four people write it up. Only one of them is right; hover over each to see what goes wrong in the others.
Annotate
On: \( \begin{aligned} &(1)\; \text{do not reject, since } 0.03 < 0.05 \\ &(2)\; \text{reject; a 3 percent chance } H_0 \text{ is true} \\ &(3)\; \text{reject; so the effect is large} \\ &(4)\; \text{reject; data this extreme: 3 in 100 if } H_0 \end{aligned} \)
Readings two and three are the dangerous ones, precisely because they reach the right decision. Nothing in the conclusion flags the faulty reasoning, so they survive into reports and news stories. Only the explanation exposes them.
The habit that protects you is to say the condition out loud every time: if the null were true, data this extreme would happen this often. A sentence about a p-value that leaves out the if has almost always gone wrong.
Worked example
A test score has standard deviation 15. A new study method raises the mean score by just 0.1 point, an effect nobody would notice. Find the right-tailed p-value for that difference with a sample of 100 and with a sample of 100,000.
The difference is the same in both studies; only the sample size changes. Watch what the sample size does to the standard error, and through it to the z-score.
Standard error with 100
Why: 15 over the square root of 100.
\[ \frac{15}{\sqrt{100}} = \frac{15}{10} = 1.5 \]
z and p-value with 100
Why: a tenth of a point is a tiny fraction of 1.5.
\[ z = \frac{0.1}{1.5} = 0.067, \quad p = P(Z \ge 0.067) = 0.473 \]
Standard error with 100,000
Why: 15 over the square root of 100,000.
\[ \frac{15}{\sqrt{100000}} = \frac{15}{316.23} = 0.0474 \]
z and p-value with 100,000
Why: now a tenth of a point is two standard errors.
\[ z = \frac{0.1}{0.0474} = 2.108, \quad p = P(Z \ge 2.108) = 0.0175 \]
Figure (svg): Two bell curves, both the null's sampling distribution of a mean difference centred at zero. Left, n equals 100, standard error 1.5, axis from minus 5 to 5: the observed difference 0.1 sits almost at the centre, and nearly half the curve, 0.47, is shaded beyond it. Right, n equals 100,000, standard error 0.047, axis from minus 0.2 to 0.2: the same 0.1 sits out in the tail, and only a red sliver, 0.0175, lies beyond it.
In context: the large study declares a significant improvement at the 5 percent level, and the improvement is a tenth of a point on a test with a standard deviation of 15. Significant means hard to explain by chance; it does not mean big, and it does not mean important.
Carry this into reading research. When a study reports a significant result, look for the size of the effect before deciding whether it matters. A huge sample makes almost any difference significant, so significance alone tells you little about how much changed.
Check the ratio of the standard errors
Why: a sample 1,000 times larger shrinks the standard error by the square root of 1,000.
\[ \frac{1.5}{0.0474} = 31.6 = \sqrt{1000} \]
Counterexample
The previous slide showed significance without importance. The reverse also happens, and it is just as easy to misread.
A small pilot study of a reading programme tests 9 pupils. The mean score rises by 8 points on a test with standard deviation 15, a large and educationally important gain. The test is right-tailed at alpha equal to 0.05.
\[ \frac{15}{\sqrt{9}} = 5, \quad z = \frac{8}{5} = 1.6, \quad p = P(Z \ge 1.6) = 0.0548 \]
Discussion prompt
The p-value is 0.0548, so the test does not reject at the 5 percent level. Does this show the programme does not work? What should the researchers say instead?
Put the two slides together and the p-value's job comes into focus. It measures how hard the data are to explain by chance, which depends on the size of the effect and on the size of the sample at once. To know how big an effect is, look at the effect itself: the 8 points, or the 0.1 point.
Trap
The tutoring pilot gives a p-value of 0.0548 and the report says:
\[ p > 0.05 \]
\[ \Rightarrow \text{ the programme has no effect} \]
The arithmetic and the decision are both right; the last line claims far more than they support.
Wrong. Not rejecting is not accepting.
The book is explicit: when you do not reject, it does not mean you should believe the null is true. The data have simply failed to provide sufficient evidence against it.
\[ \text{do not reject } H_0 \]
\[ = \text{ not sufficient evidence for } H_a \]
A small sample can fail to detect a large effect. Absence of evidence, from 9 pupils, is not evidence of absence.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the third bullet of the decision rule
Discrimination
A right-tailed test of whether a new fertiliser increases mean yield gives a p-value of 0.02. Sort each statement. The test for every one: does it keep the condition that the null is true, and does it claim only what a p-value can say?
Sort into buckets
Sort each statement about the p-value of 0.02.
The statement about a 98 percent chance deserves a second look, because it sounds modest and careful. It is the same error as the 2 percent chance the fertiliser does nothing, just written as a complement. Neither number exists in the test.
A correct sentence about a p-value has a recognisable shape: it starts with if the null were true, or ends with under the null. When you write one yourself, check for that clause.
Matching
Every common misreading of a p-value is broken by one specific fact about how it was computed. Match them.
Match the pairs
Why: A probability computed by assuming the null cannot measure the null. A large p-value says the data are ordinary under the null, which does not single the null out. Smallness of p mixes effect size with sample size. And two p-values on the same side of alpha can differ in strength by a factor of forty.
These four facts are worth more than a list of forbidden sentences. A new misreading you have never seen before will almost always break against one of them.
The last pair leads into Part 5. The decision rule sorts results into two bins, but the book itself warns against stopping there: a data analyst should have more confidence in a rejection at 0.001 than one at 0.04, even with alpha at 0.05.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — Thinking about the meaning of the p-value
Section
Part 5
Concept
After you make your decision, the book asks you to write a thoughtful conclusion about the hypotheses in terms of the given problem. Reject H0 is not a conclusion; it is a step. The conclusion is a sentence someone who never saw your arithmetic could act on.
\[ \text{At the } \alpha \text{ level, there is (not) sufficient evidence that } \langle H_a \text{ in words} \rangle \]
The template is always about the alternative. When you reject, the evidence is sufficient for it; when you do not reject, the evidence is not sufficient for it. You never conclude the null is true, and you never say the alternative is proved.
Notice that the conclusion is about the population. The sample mean of the baker's ten loaves was 17 cm; that is a fact, and needs no test. The claim the test speaks to is about all the loaves he bakes.
The word not is the only thing that changes between the two decisions. The level, the word evidence and the claim stay fixed, which makes the conclusion almost mechanical once the decision is made.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Conclusion: write a thoughtful conclusion in terms of the given problem
Worked example
The baker's test rejected the null at the 5 percent level. Build the conclusion piece by piece, then check it would make sense to one of his doubtful customers.
Each step adds one part of the template from the previous slide. Write them in order and the sentence assembles itself.
The level
Why: the standard that was applied, fixed before the data.
\[ \text{At the 5 percent significance level,} \]
The evidence
Why: the test rejected, so the evidence was sufficient.
\[ \text{there is sufficient evidence} \]
The claim, in the alternative's direction
Why: greater than 15.
\[ \text{that the mean height is greater than } 15 \text{ cm} \]
The population
Why: all the loaves he bakes, not the ten in the sample.
\[ \text{for the population of the baker's loaves.} \]
Put together, this is almost word for word the book's own conclusion: there is sufficient evidence that the true mean height for the population of the baker's loaves of bread is greater than 15 cm.
Now read it as one of the doubtful customers would. It tells them what standard was used, what the evidence showed and what it was about, with no z-score in sight. That is the test of a good conclusion: someone who never saw the working could act on it.
Check the sentence for forbidden words
Why: it says sufficient evidence, not proved; it is about the population mean, not the sample; and it names the level.
\[ \text{no proof, no acceptance, no missing level} \]
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9's conclusion
Worked example
A café's machine is set to fill cups with 8 ounces on average; fills are normal with sigma equal to 0.4 ounce. The owner suspects it is under-filling. A random sample of 25 cups averages 7.85 ounces. Test at the 5 percent level and write the conclusion.
A full test, start to finish. The suspicion is that the mean is less than 8, so the alternative points left and so will the tail.
Hypotheses
Why: under-filling means less than 8; the null takes the complement.
\[ H_0: \mu \ge 8, \qquad H_a: \mu < 8 \]
Standard error
Why: sigma is known and fills are normal: a z-test.
\[ \frac{0.4}{\sqrt{25}} = \frac{0.4}{5} = 0.08 \]
Standardise
Why: the sample sits below the null value.
\[ z = \frac{7.85 - 8}{0.08} = \frac{-0.15}{0.08} = -1.875 \]
The left tail
Why: the alternative points left.
\[ p\text{-value} = P(Z \le -1.875) = 0.0304 \]
Figure (svg): A bell curve centred at 8 ounces with standard error 0.08, axis from 7.7 to 8.3 with ticks every standard error. A red dashed line marks the sample mean 7.85, labelled z equals minus 1.875, and the left tail below it is shaded red and labelled p equals 0.0304.
Decide
Why: alpha is greater than the p-value.
\[ 0.05 > 0.0304 \;\Rightarrow\; \text{reject } H_0 \]
Conclusion: at the 5 percent level there is sufficient evidence that the machine's mean fill is less than 8 ounces. Note what it does not say: it does not say the machine fills 7.85 ounces, only that the 8-ounce setting is no longer believable.
At the 1 percent level the same p-value of 0.0304 would not reject. So before calling the engineer, the owner should have decided which mistake is worse: adjusting a machine that was fine, or leaving a short-filling machine in service.
Check with the cutoff
Why: at 0.05 a left-tailed test rejects below 8 minus 1.645 standard errors, and 7.85 is below it.
\[ 8 - 1.645(0.08) = 7.868 > 7.85 \]
Worked example
A tutoring centre claims that more than half of its students raise their grade within a term. In a random sample of 100 students, 56 did. Test the claim at the 5 percent level and write the conclusion.
A test of a proportion, from Section 9.3's third row. The conditions need checking first, then the same four moves as a mean.
Hypotheses
Why: the claim is more than half, so it goes in the alternative.
\[ H_0: p \le 0.50, \qquad H_a: p > 0.50 \]
Check the normal approximation
Why: both np and nq must exceed five.
\[ np = 100(0.5) = 50 > 5, \quad nq = 50 > 5 \]
Standard error under the null
Why: use the null's p, not the sample's.
\[ \sqrt{\frac{pq}{n}} = \sqrt{\frac{0.5 \times 0.5}{100}} = \sqrt{0.0025} = 0.05 \]
Standardise and take the right tail
Why: the sample proportion is 0.56.
\[ z = \frac{0.56 - 0.50}{0.05} = 1.2, \quad p\text{-value} = P(Z \ge 1.2) = 0.1151 \]
Figure (svg): A bell curve centred at 0.50 with standard error 0.05, axis from 0.33 to 0.67. A yellow dashed line marks the sample proportion 0.56, z equals 1.2, and the right tail beyond it is shaded blue and labelled p equals 0.1151, a visibly substantial area.
Decide
Why: alpha is at or below the p-value.
\[ 0.05 < 0.1151 \;\Rightarrow\; \text{do not reject } H_0 \]
Conclusion: at the 5 percent level there is not sufficient evidence that more than half of the centre's students raise their grade within a term. It does not say that half or fewer do; 56 out of 100 is simply not unusual enough for a fifty-fifty rate to be doubted.
Notice the standard error used the null's proportion, 0.5, not the sample's 0.56. The curve describes the imaginary world in which the null is true, so every number that builds it comes from the null; the sample only supplies the boundary of the tail.
Check the p-value against the empirical rule
Why: 1.2 standard errors is a little past 1, where the one-sided tail is about 0.16, so the answer should be a little below 0.16.
\[ 0.1151 < 0.16 \]
Picture it
The book's Section 9.5 adds a note about the meaning of the p-value that is easy to skip and worth keeping. Here are four p-values against a fence at 0.05, on a scale where each step is ten times smaller.
Figure (svg): A logarithmic number line of p-values from 0.0005 to 1, with a yellow dashed fence at alpha 0.05. Red dots at 0.001 and 0.04 lie left of the fence and are bracketed together as same decision, 40 times apart. Blue dots at 0.056 and 0.4 lie right of it. A pink bracket joins 0.04 and 0.056 across the fence: nearly the same evidence, opposite decisions.
The red pair, 0.001 and 0.04, land on the same side of the fence, yet one is forty times smaller than the other. The book says an analyst should have more confidence in the rejection at 0.001.
The pink pair, 0.04 and 0.056, land on opposite sides, yet they are almost the same evidence. Likewise a p-value of 0.4 gives much more confidence in not rejecting than 0.056 does. This, in the book's words, makes the analyst use judgement rather than mindlessly applying rules.
So report both: the decision at the preset level, which is what the procedure promised, and the p-value itself, which tells the reader how close the call was. A reader with a stricter or looser standard than yours can then make their own decision from your report.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — Thinking about the meaning of the p-value
Sorting
Every sentence below reports the coffee-machine test, which rejected the null at the 5 percent level with a p-value of 0.0304. Sort them. An acceptable sentence names the level, speaks of evidence and describes the population mean in context.
Sort into buckets
Sort each sentence about the coffee-machine test.
The sentence that gives 7.85 ounces as the machine's mean is worth a pause. That is the sample's mean, a fact about 25 cups. The test makes no claim about where the population mean is, only that 8 is no longer believable.
And the bare Reject H0 is not wrong, merely unfinished. It is the decision, the fifth box in the procedure; the conclusion is the sixth.
Step zero
A conclusion has to use the problem's own words, so before computing anything, collect them. Here is a new claim to practise on.
A city says its buses arrive late on fewer than 10 percent of trips. A rider's group samples 400 trips at random and finds 52 late ones. The group will test at alpha equal to 0.01.
Discussion prompt
Before any arithmetic: what is the parameter in words, what does the alternative say in words, what level will the conclusion name, and which tail will the p-value be?
Doing this first protects the conclusion from the most common failure: a correct calculation reported in words that do not match the question. The sample proportion here is 52 over 400, which is 0.13, but that number belongs in the working, not in the claim.
If you carry the test through, the standard error is 0.015, z is 2, and the p-value is 0.0228, so at the 1 percent level the riders do not have sufficient evidence, even though they would at 5 percent.
Comparison
Each row is a test from this deck. Fill in the decision and whether the evidence is sufficient for the alternative.
Comparison matrix
| test | Hₐ | α | p-value | decision | evidence for Hₐ |
|---|---|---|---|---|---|
| baker | μ > 15 cm | 0.05 | ≈ 0 | reject | sufficient |
| Try It 9.9 | μ > 12 | 0.001 | 0.00135 | do not reject | not sufficient |
| Genetics Labs | p > 0.50 | 0.01 | 0.025 | do not reject | not sufficient |
| coffee machine | μ < 8 oz | 0.05 | 0.0304 | reject | sufficient |
| tutoring centre | p > 0.50 | 0.05 | 0.1151 | do not reject | not sufficient |
Read down the decision column against the last two numbers. Every reject has alpha strictly greater than the p-value; every do-not-reject has alpha at or below it. The evidence column then follows the decision automatically.
Now read across the Try It 9.9 row. Its p-value is the second smallest in the table, yet it is a do-not-reject, because its alpha is the strictest. A p-value is only small or large relative to a level.
Section
Part 6
Pattern
Every test in this deck ran through the same six steps. The arithmetic lives in steps 3 and 4; the marks are usually lost in steps 1, 5 and 6.
Figure (svg): A flow of six boxes. 1 hypotheses, from the claim. 2 assume H0: draw its sampling distribution. 3 locate the sample: z equals observed minus null over standard error. 4 p-value: the tail the alternative points to. 5 decide: alpha greater than p, reject; otherwise do not. 6 conclude: level, evidence, claim in context.
Use the list as a checklist on your own work. Each line names a decision, and most wrong answers in this section come from making one of them silently: a tail chosen without reading the alternative, a level chosen after seeing the p-value, a conclusion written about the sample.
\[ p\text{-value} = P(\text{as extreme or more} \mid H_0), \qquad \alpha > p \Rightarrow \text{reject} \]
Two habits hold the pattern together. Fix the hypotheses and alpha before you look at the sample. And keep the condition that the null is true attached to every sentence you write about a p-value.
Check
Pick the sentence that keeps the condition in the right place.
Before looking at the options, say the definition to yourself, including the if clause.
Check your understanding
A test of whether a new battery lasts longer than the old one gives a p-value of 0.01. Which statement is correct?
Answer: A
Why: The p-value is computed by assuming the null is true, here that the new battery lasts no longer, and it gives the chance of a sample at least as extreme as the one observed. It says nothing directly about the probability of either hypothesis.
If you chose correctly, push yourself one step further: rewrite the correct option for a p-value of 0.30. It still begins with the same if clause; only the frequency changes.
Two of the wrong options are the same mistake in different clothes: one gives the probability of the null, the other the probability of the alternative. Both swap the two sides of the given-that bar.
The option about a much longer life confuses significance with size. To know how much longer the new battery lasts, look at the difference in mean lifetimes, not at the p-value.
Check
Compare, then decide; then make sure the words match the decision.
Write the comparison as an inequality with alpha on the left before reading the options.
Check your understanding
A test is run at the 1 percent level and gives a p-value of 0.018. What is the correct decision and conclusion?
Answer: A
Why: Alpha, 0.01, is less than the p-value, 0.018, so the rule says do not reject. The conclusion is that the evidence is not sufficient for the alternative at the 1 percent level, nothing stronger.
Read the inequality aloud as a sentence: alpha, one in a hundred, is smaller than the p-value, almost two in a hundred, so the sample is not rare enough to meet this test's standard.
At the 5 percent level the same p-value would reject, which is why the conclusion has to name its level. Without it, a reader cannot tell whether 0.018 passed or failed.
One wrong option reaches the right decision and then says too much. That pattern, right decision with wrong meaning, is the one to watch for in your own writing.
Check
This one needs the whole procedure. Sketch the curve before choosing.
Find the standard error first, then the z-score, then decide which tail the alternative points to.
Check your understanding
A bottler claims its bottles hold more than 500 mL on average. Fills are normal with sigma 4 mL. A sample of 16 bottles averages 502 mL. At alpha 0.05, what are the p-value and conclusion?
Answer: A
Why: The standard error is 4 over the square root of 16, which is 1. So z is 2 over 1, which is 2, and the right tail beyond 2 is 0.0228. Since 0.05 is greater than 0.0228, reject: there is sufficient evidence at the 5 percent level that the mean fill exceeds 500 mL.
In context: if the bottles really averaged 500 mL, a sample of 16 averaging 502 mL or more would turn up only about 23 times in 1,000, so the claim of more than 500 mL is supported at the 5 percent level.
\[ \frac{4}{\sqrt{16}} = 1, \quad z = \frac{502 - 500}{1} = 2, \quad P(Z \ge 2) = 0.0228 \]
Each wrong option is a different step of the pattern done wrongly: the standard error, the tail, and the conclusion's wording. If you missed this one, find which step and redo just that step.
Explain it to yourself
The whole section rests on one piece of reasoning. Explaining it without symbols is the best test of whether it has landed.
Discussion prompt
Explain, in three or four sentences and without any formulas, why a small p-value is a reason to reject the null hypothesis, but a large p-value is not a reason to accept it.
A strong answer names the condition: the probability was computed as if the null were true. A very strong one names the asymmetry: unlikely data rule an assumption out, but likely data rule nothing in.
If your answer used the words proves or accepts anywhere, rewrite that sentence. The strongest words a test earns are sufficient evidence and doubt.
Exit ticket
A city reports that the mean commute is 25 minutes, with a standard deviation of 6 minutes. A transport researcher thinks commutes have become longer. A random sample of 36 commuters averages 27.1 minutes. The level is 0.05.
Discussion prompt
State the hypotheses, find the p-value, make the decision, and write the conclusion in context.
Check yourself against the pattern. The alternative came from the word longer. The standard error was 1 because 6 divided by the square root of 36 is 1. The tail was the right one. The decision compared alpha with the p-value in the right order, and the conclusion named the level and the population mean.
If your p-value was 0.0357, you shaded both tails; if it was near 0.36, you used the standard deviation of single commutes instead of the standard error of the mean.
Recap
A hypothesis test runs on the rare-event argument: assume the null, and if the sample would be very unlikely under that assumption, doubt the assumption. The p-value measures how unlikely, and a preset alpha turns the measurement into a decision.
| idea | statement | read it as |
|---|---|---|
| rare-event argument | unlikely under H₀ ⇒ doubt H₀ | the data are real; H₀ is only an assumption |
| p-value | P(as extreme or more | H₀ true) | a tail area on the null's curve |
| direction | Hₐ: < left, > right, ≠ both tails | the alternative chooses the tail |
| decision rule | α > p-value ⇒ reject H₀ | α ≤ p-value ⇒ do not reject |
| alpha | set before the data; 0.05 if none given | P(Type I error) = α |
| not | P(H₀ | data), effect size, proof | conditional on H₀; significant ≠ important |
| conclusion | at level α, (not) sufficient evidence that Hₐ | in context, about the population |
The arithmetic was Chapter 7's: a standard error, a z-score, a tail area. The new skills are the logic around it: keeping the if-the-null-is-true condition attached to every p-value, comparing with alpha in the right direction, and writing a conclusion that claims no more than the evidence allows.
Section 9.5 practises full tests of all three shapes, left, right and two-tailed. The Introductory Business Statistics text works the same procedure in its §9.4, with business examples, if you want more practice in a different setting.
OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-469 — the whole section
OpenStax Introductory Business Statistics 2e, §9.4 Full Hypothesis Test Examples §9.4 Full Hypothesis Test Examples — the same procedure in business contexts
Want this taught 1-on-1? Alexander tutors Statistics — $55/session, free consultation.