9.4 Rare Events, the Sample, Decision and Conclusion

The section that supplies the logic behind a hypothesis test. The reasoning is the rare-event argument: make an assumption about the population, gather data, and if the sample has properties that would be very unlikely to occur when the assumption is true, conclude that the assumption is probably incorrect. The p-value measures that unlikeliness — it is the probability that, if the null hypothesis is true, another randomly selected sample would give results as extreme or more extreme than the ones obtained. A large p-value indicates the null should not be rejected, and the smaller the p-value the stronger the evidence against it. Comparing the p-value against a significance level chosen before the data were collected turns the measurement into a decision, and the conclusion must then be written in the words of the problem. The p-value is conditional on the null being true, which is what makes almost every common misreading of it wrong.

Subject: Statistics · 67 slides · symbolic lesson

Open the interactive version of this deck

What this lesson covers

The lesson, slide by slide

1. Rare Events, the Sample, Decision and Conclusion

Title

Statistics · Chapter 9 — Hypothesis Testing with One Sample

How surprising is the sample, if the null were true, and what may you say about it?

2. What this lesson gives you

Objectives

Sections 9.1 to 9.3 set up the pieces of a test: a pair of hypotheses, the two errors you might make, and the distribution to use. None of that yet tells you how to decide. This section supplies the reasoning that turns a sample into a decision, and the sentence that reports it.

The arithmetic in this section is the arithmetic of Chapter 7: a standard error, a z-score, an area under a normal curve. What is new is the logic around it. Most marks lost on hypothesis tests are lost in the logic, not the arithmetic, so the logic gets most of the slides.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-469 — the section this deck covers

3. Before anything new: a sample mean in the tail

Warm-up

Chapter 7 gave you the sampling distribution of a mean. Use it on a question that will turn out to be this whole section in disguise.

Discussion prompt

A population has mean 50 and standard deviation 10. You take a random sample of 25. What is the probability that the sample mean is 54 or more?

Now turn the question round. Suppose someone claims the population mean is 50, and your sample of 25 comes back with a mean of 54. You have just computed how surprising that sample would be if the claim were true. That probability, 0.0228, is a p-value, and the rest of this lesson is about what you may and may not conclude from it.

OpenStax Introductory Business Statistics 2e, §9.4 Full Hypothesis Test Examples §7.2, the Central Limit Theorem for sample means — the standard error used here

4. The rare-event argument

Section

Part 1

5. Assume, look, and doubt

Concept

Every hypothesis test in this book runs on one piece of reasoning. You make an assumption about the population: that assumption is the null hypothesis. Then you gather a random sample.

If the sample has properties that would be very unlikely to occur if the assumption were true, you conclude that the assumption is probably incorrect.

Figure (svg): A flow diagram of three boxes joined by arrows. Box 1, assume: the null hypothesis is true. Box 2, compute: how likely is a sample at least this extreme, if the assumption holds? Box 3, judge, which splits into two exits: a very small probability leads to a red box, doubt the assumption, reject H0; a probability that is not small leads to a grey box, no verdict, do not reject H0.

Only one exit says anything. A small probability turns a guess into a doubtful guess; a large one leaves the guess exactly where it was.

Look at the two exits from the last box. Only the left one says anything. A very unlikely sample gives you grounds to doubt the assumption. A sample that would be quite ordinary under the assumption gives you no verdict at all, because ordinary samples are also what many other assumptions would produce.

rare-event argument — If a sample would be very unlikely when an assumption about the population is true, the assumption is probably incorrect. The probability is always computed as if the assumption held.

Keep the book's parenthesis in mind as you go. Your assumption is just an assumption; it may or may not be true. Your sample data are real, and they are showing you a fact that seems to contradict the assumption. When a guess and a fact disagree, the guess gives way.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Rare Events

6. Didi and Ali: one hundred-dollar bill among 200 bubbles

Worked example

At a wealthy friend's birthday party, Didi and Ali are told there are 200 plastic bubbles in a tall basket, and only one holds a hundred-dollar bill. Blindfolded, Didi reaches in first and pulls out a bubble. It contains the bill. What should Ali make of that?

Run the argument in its three moves. First write down the assumption you were given, then work out how likely Didi's draw is if that assumption is true, then judge the number.

State the assumption

Why: what they were told plays the part of the null hypothesis.

\[ \text{assumption: exactly one bill among } 200 \text{ bubbles} \]

Count the ways the observed event can happen

Why: one bubble out of 200 equally likely ones holds the bill.

\[ P(\text{first bubble holds the bill}) = \frac{1}{200} \]

Write it as a decimal

Why: so it can be judged against other probabilities.

\[ \frac{1}{200} = 0.005 \]

Judge it

Why: five chances in a thousand is a rare event.

\[ 0.005 \text{ is very small} \]

So Ali doubts what they were told and hopes there are more bills in the basket. Notice what he doubts: not the draw, which he watched happen, but the story about the basket, which nobody has checked. That is the rare-event argument exactly.

Notice that the probability was computed entirely inside the story they were told. Nobody needs to know how many bills are really in the basket to get 0.005; you only need the assumption. That is what makes the calculation possible, and it is why the answer is a verdict on the assumption.

Check the probability by counting parties

Why: over 200 such parties with one bill in the basket, the first draw should win about once, and lose the rest of the time.

\[ 200 \times 0.005 = 1, \qquad 200 \times 0.995 = 199 \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Didi and Ali

7. Which baskets would make Didi's draw ordinary?

Picture it

Ali's doubt has a direction: he suspects more bills, not fewer. The chart shows why. Each bar is the probability that the first bubble holds a bill, for a different number of bills in the basket of 200.

Figure (svg): A bar chart. Horizontal axis: the number of hundred-dollar bills in a basket of 200 bubbles, at 1, 2, 5, 10, 20 and 50. Vertical axis: the probability that the first bubble drawn holds a bill, k over 200. The bar for 1 bill is red and tiny, 0.005. The bars rise through 0.01, 0.025, 0.05 and 0.10 to 0.25 for 50 bills.

Under the story they were told, Didi's draw is a one-in-two-hundred event. Under almost any other story it is far less surprising, which is why the story is what Ali starts to doubt.

Under the story they were told, one bill, Didi's draw has probability 0.005: the red bar is barely visible. With ten bills it would be 0.05, with fifty it would be 0.25, a one-in-four event that would surprise nobody.

\[ P(\text{first bubble holds a bill}) = \frac{k}{200} \]

This is the comparison hiding inside every test. The observed fact fits the stated assumption badly and fits other assumptions much better. The test never proves which other assumption is right; it only says that the stated one has been made to look doubtful.

It also shows the limits. Even with fifty bills in the basket, three draws in four would come up empty. So a single lucky draw cannot tell Ali how many bills there are, only that one bill is a poor explanation of what he saw.

8. The assumption is a guess; the data are a fact

Intuition

Why should the assumption give way, rather than the data? Because they have different standing. The null hypothesis is something you wrote down before looking. The sample is something that actually happened.

\[ \text{unlikely under } H_0 \;+\; \text{the data are real} \;\Longrightarrow\; \text{doubt } H_0 \]

Think of a friend who swears a coin is fair, and then it lands heads ten times running. You do not decide that the ten heads did not happen. You decide that the friend's claim about the coin is in trouble.

The same asymmetry runs the other way. If the coin had landed heads six times out of ten, you would not decide the coin is fair. Six heads is what a fair coin produces, and also what a slightly biased coin produces, so it settles nothing.

That is why the decisions in Section 9.1 were worded the way they were: reject the null, or do not reject it. There is no third option called accept, because ordinary data cannot single out the null from its neighbours.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — the parenthesis on assumptions and data

9. Ten heads in a row

Prediction

A friend hands you a coin and says it is fair. You flip it ten times and get ten heads. Commit to an answer before you work anything out.

Ask yourself which of the two things you have is a fact and which is a claim, then decide which should give way.

Predict first

What is the most reasonable conclusion?

  • The claim that the coin is fair is now doubtful
  • The coin is fair, and you were simply lucky
  • The coin is certainly biased toward heads
  • Nothing can be concluded from only ten flips

Correct: The claim that the coin is fair is now doubtful

Why: Ten heads from a fair coin has probability one in 1,024. That is rare enough to cast serious doubt on the claim of fairness, but rare is not impossible, so the honest word is doubtful rather than certainly biased.

Two of the wrong answers are the two ways people mishandle evidence. One refuses to let any data move the assumption. The other treats a rare event as an impossible one and leaps to certainty.

The rare-event argument sits between them: it lets the data count against the claim, in proportion to how unlikely the data would be if the claim were true. The next slide puts a number on it.

10. The deck's own example: ten heads from a coin said to be fair

Worked example

A coin is claimed to be fair. Ten flips give ten heads. Find the probability of this result if the claim is true, and decide whether the claim is in doubt.

The flips are independent, so the probability of a run of heads is a product of ten one-halves. Write the product before evaluating it, so you can see where the number comes from.

State the assumption

Why: fair means each flip lands heads with probability one half.

\[ H_0: p = 0.5 \]

Multiply across independent flips

Why: ten heads in a row means heads on every one.

\[ P(\text{10 heads}) = \left(\tfrac{1}{2}\right)^{10} \]

Evaluate the power

Why: two to the tenth is 1,024.

\[ \left(\tfrac{1}{2}\right)^{10} = \frac{1}{1024} \]

Write it as a decimal

Why: about one in a thousand.

\[ \frac{1}{1024} \approx 0.000977 \]

In context: if the coin really were fair, ten heads in ten flips would happen about once in every thousand sets of ten. That is rarer than Didi's draw, so the claim of fairness is in more serious doubt than the story about the basket was.

Compare the rare events met so far. Didi's draw was 1 in 200; four sixes in a row on a fair die would be about 1 in 1,300; ten heads is about 1 in 1,000. None is impossible, and each is rare enough that most people would stop trusting the claim behind it.

Check against the binomial distribution

Why: the binomial formula with n equal to 10 and x equal to 10 must give the same number.

\[ \binom{10}{10}(0.5)^{10}(0.5)^{0} = 1 \times \frac{1}{1024} \times 1 = 0.000977 \]

11. Every exact sequence is rare

Counterexample

Here is a puzzle that breaks a careless version of the argument. Flip a fair coin ten times and write down the exact sequence. Suppose it comes out heads, tails, heads, heads, tails, tails, heads, tails, heads, tails.

\[ P(\text{HTHHTTHTHT}) = \left(\tfrac{1}{2}\right)^{10} = 0.000977 \]

That sequence is exactly as unlikely as ten heads. Yet nobody would doubt the coin after seeing it.

Discussion prompt

If every exact sequence of ten flips has probability 0.000977, why does ten heads cast doubt on the coin while this mixed sequence does not?

Figure (svg): A probability histogram for the number of heads in 10 flips of a fair coin, bars at 0 to 10, symmetric and peaking at 5 with height 0.246. The bars at 9 and 10 heads are red; the bar at 10 is labelled 0.001 and the pair is labelled 9 or more: 0.0107.

Count heads and the tail carries the evidence: 9 or 10 heads together happen about once in a hundred tries. A single exact sequence of flips cannot do that job, because every sequence is one in 1,024.

This is why the p-value, coming up in Part 2, is defined with the words as extreme or more extreme. A single exact outcome is almost always improbable. A tail of outcomes, all pointing away from the null in the same direction, is what measures surprise.

12. Trap: rare read as impossible

Trap

The trap

After Didi's draw, a first attempt at Ali's conclusion says:

\[ P = 0.005 \]

\[ \Rightarrow \text{ there must be more bills} \]

The probability is right. The word must is the error: it treats a one-in-two-hundred event as if it could not happen.

Too strong. Rare events do happen.

The fix

With one bill in the basket, the first draw still wins once in 200 parties. Didi may simply have been lucky. What the argument gives is grounds for doubt, not proof.

\[ P = 0.005 \]

\[ \Rightarrow \text{ the one-bill story is doubtful} \]

That is also why every test carries a chance of a Type I error from Section 9.2: rejecting an assumption that was true all along, exactly when the rare event happens.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Didi and Ali

13. Does the evidence cast doubt on the assumption?

Sorting

For each situation, compute or estimate how likely the observation would be if the stated assumption were true. Then decide whether it casts doubt on the assumption or gives no verdict.

Sort into buckets

Sort each observation by what it does to the stated assumption.

rare under the assumption: doubt it
a die said to be fair shows a six on four rolls in a row (probability 0.00077); a basket said to hold 1 winning bubble in 200 gives a winner on the first draw; a lottery said to be random draws the same five numbers two weeks running
ordinary under the assumption: no verdict
a coin said to be fair gives 6 heads in 10 flips; a machine said to average 8 ounces fills 25 cups averaging 7.98 ounces (SE 0.08); a class said to average 70 scores a sample mean of 71 with SE 2
doubt
Each would happen well under one time in a hundred if the assumption held, so the assumption, not the observation, comes under suspicion.
none
Six heads, a mean a quarter of a standard error low, a mean half a standard error high: each is what the assumption predicts, so it cannot count against it, and cannot prove it either.

Look at the three in the no-verdict bucket. None of them confirms its assumption; a slightly unfair coin or a machine averaging 7.95 ounces would produce the same data just as easily. Ordinary data are consistent with the assumption and with its neighbours.

Notice too that the two means needed a standard error before they could be judged. A mean of 7.98 ounces sounds low until you see it is only a quarter of a standard error below 8. From here on, every judgement of rarity goes through the sampling distribution.

14. The p-value

Section

Part 2

15. The p-value: how surprising, if the null is true

Concept

Part 1 judged rarity by eye. A test needs a number, and the book names it. Use the sample data to calculate the probability of getting the test result, called the p-value.

p-value — The probability that, if the null hypothesis is true, the results from another randomly selected sample will be as extreme or more extreme as the results obtained from the given sample.

\[ p\text{-value} = P(\text{a result at least as extreme as yours} \mid H_0 \text{ true}) \]

Read the vertical bar as given that. The whole calculation lives in an imaginary world where the null hypothesis is true. You build that world's sampling distribution, place your sample in it, and measure the area beyond.

Then read the size of the number. A large p-value means your sample is the kind of thing the null produces routinely, so you should not reject the null. The smaller the p-value, the more unlikely the outcome, and the stronger the evidence against the null.

The book adds a piece of advice you should take literally: draw a graph that shows the p-value. The test is easier to perform with a picture, because you see the problem more clearly, and every worked example in this deck has one.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — Using the Sample to Test the Null Hypothesis

16. Reading the p-value piece by piece

Notation

Here is the p-value for a right-tailed test of a mean, the kind in Example 9.9. Every symbol in it carries part of the definition, so decode each one before using it.

Annotate

On: \( p\text{-value} = P(\bar{X} \ge \bar{x}_{\text{obs}} \mid \mu = \mu_0) \)

  • The random sample mean: what another randomly selected sample would give. It has a whole distribution.
  • Your sample's mean: one fixed number, already observed. It is the boundary of the tail.
  • As extreme or more extreme. The direction comes from the alternative: ≥ for a right-tailed test, ≤ for a left-tailed one.
  • Given the null is true. The curve is centred at the null value, not at your sample mean. This condition is what every misreading forgets.

The capital and small letters matter here. The capital X-bar is the random quantity that varies from sample to sample; the small x-bar with its label is the one value you actually got. The p-value asks how often the random one would land beyond the fixed one.

And the condition after the bar is not decoration. Take it away and the sentence stops being something you can compute, because without assuming a value for the mean there is no curve to find an area under.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-468 — the definition and Example 9.9

17. Why a tail, and not the single value you observed

Concept

You might ask why the p-value includes results more extreme than yours. Why not just the probability of the sample you got?

The first reason is the coin from Part 1: an exact outcome is nearly always improbable, so its probability cannot measure surprise. The second is sharper. For a continuous variable such as a sample mean, the probability of any one exact value is zero.

\[ P(\bar{X} = 17) = 0 \quad \text{for every possible population mean} \]

A probability that is zero whatever the truth is cannot count against anything. So the book's definition takes the area of the whole tail beyond your result: all the samples that would be at least as surprising as yours.

\[ P(\bar{X} \ge 17 \mid \mu = 15) \]

Read that tail as a ranking. Your sample sits at some point on the null's curve. The p-value is the fraction of all possible samples that would rank as at least as extreme. A p-value of 0.01 says only one sample in a hundred would rank as far out as yours.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the p-value as an area beyond the sample mean

18. Example 9.9: the baker's hypotheses and the null curve

Worked example

A baker claims that his bread height is more than 15 cm, on average. Several customers do not believe him. He bakes 10 loaves; their mean height is 17 cm. From baking hundreds of loaves he knows the standard deviation of height is 0.5 cm and the heights are normal. Set up the test and the null's sampling distribution.

This slide does the setting up; the next one does the probability. The order matters, because the curve you draw depends on the hypotheses you wrote, not on the sample.

Write the alternative from the claim

Why: is more than translates as greater than, and the claim the baker wants to show goes in the alternative.

\[ H_a: \mu > 15 \]

Write the null as its complement

Why: the null must contradict the alternative and carry the equality.

\[ H_0: \mu \le 15 \]

Choose the distribution

Why: sigma is known and heights are normal, so the sample mean is exactly normal: a z-test, Section 9.3's first row.

\[ \bar{X} \sim N\left(\mu, \tfrac{\sigma}{\sqrt{n}}\right) \]

Compute the standard error

Why: sigma over the square root of the sample size.

\[ \frac{\sigma}{\sqrt{n}} = \frac{0.5}{\sqrt{10}} = \frac{0.5}{3.1623} = 0.1581 \]

Centre the curve at the null's boundary value

Why: assume the null is true at its edge, a mean of 15.

\[ \bar{X} \sim N(15,\; 0.1581) \]

In context: if the baker's loaves really average 15 cm, the means of samples of 10 loaves pile up around 15 with a standard error of about 0.16 cm. That is the imaginary world in which the next slide asks how surprising 17 cm would be.

Notice that the sample mean, 17, has not been used yet. The null curve is built from three things only: the null value, sigma and the sample size. If you ever centre the curve at your own sample mean, every p-value you get from it will be about one half, and the test will have asked nothing.

Check the standard error by undoing the division

Why: multiplying back by the square root of 10 must return sigma.

\[ 0.1581 \times 3.1623 = 0.5000 = \sigma \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-468 — Example 9.9

19. Why the null curve sits exactly at the boundary

Concept

The baker's null says the mean is at most 15, a whole range of values. Yet the curve was centred at exactly 15. Why that one value, and not 14.8 or 14?

Because the boundary is the null's best case. Any mean below 15 would slide the curve further left and make the observed sample even more surprising. Try It 9.9, coming up shortly, shows it with numbers you can see.

\[ \mu = 12: \quad z = \frac{12.5 - 12}{1/6} = 3, \quad p = 0.00135 \]

\[ \mu = 11.9: \quad z = \frac{12.5 - 11.9}{1/6} = 3.6, \quad p = 0.00016 \]

Moving the assumed mean down by a tenth makes the p-value about eight times smaller. So the p-value computed at the boundary is the largest one any value in the null could give. If even that is small, every version of the null is in trouble at once.

That is why the equality sits in the null and the calculation happens at the boundary. It is the most generous reading of the null, which keeps the test fair: the null is rejected only when its strongest version fails.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the distribution of sample means centred at μ = 15

20. Example 9.9: the baker's p-value

Worked example

With the null curve in hand, N(15, 0.1581), find the p-value for the observed mean of 17 cm and say what it means.

The alternative points right, so more extreme means larger. The p-value is the area of the null's curve to the right of 17. Turn 17 into a z-score first, so the area can be read from the standard normal.

Write the p-value as a tail

Why: the probability another sample mean is the same or greater than 17 when the population mean is 15.

\[ p\text{-value} = P(\bar{X} \ge 17 \mid \mu = 15) \]

Standardise the boundary

Why: the distance from the null value, in standard errors.

\[ z = \frac{17 - 15}{0.1581} = \frac{2}{0.1581} \]

Divide

Why: 12.65 standard errors above the centre.

\[ z = 12.65 \]

Read the right tail

Why: far beyond any table; node puts it near 6 times 10 to the minus 37.

\[ p\text{-value} = P(Z \ge 12.65) \approx 5.7 \times 10^{-37} \approx 0 \]

Figure (svg): A narrow bell curve centred at 15 cm, the sampling distribution of the mean of 10 loaves if the null were true, on an axis from 14.4 to 17.3. The curve rises and falls between about 14.5 and 15.5 and lies flat on the axis beyond that. A red dashed line marks the observed mean 17, far to the right, labelled 12.6 standard errors out.

The shaded tail beyond 17 is there but too thin to draw. That invisibility is the p-value: about 6 in 10 to the 37th.

In context: if the baker's loaves really averaged 15 cm, essentially no sample of 10 loaves would have a mean as high as 17 cm by chance alone. The outcome is so unlikely under the null that the evidence is strongly against it.

A number like 5.7 times 10 to the minus 37 is hard to picture. If every person on Earth had baked a batch of ten loaves every second since the universe began, you would still not expect a single batch averaging 17 cm. The book's approximately zero is exactly right.

Check the size with the three-standard-error band

Why: almost every sample mean lies within 3 standard errors of the centre, and 17 is far outside that band.

\[ 15 + 3(0.1581) = 15.47 \ll 17 \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9 and Figure 9.2

21. One loaf against the mean of ten

Picture it

Two centimetres above 15 does not sound dramatic. Why is the p-value so absurdly small? The picture puts the two relevant curves on one axis.

Figure (svg): Two bell curves centred at 15 cm on one axis from 13 to 17.6. The wide orange curve, single loaves with standard deviation 0.5, reaches out to about 16.5. The tall narrow blue curve, means of 10 loaves with standard error 0.158, is confined between about 14.5 and 15.5. A red dashed line marks 17 cm.

Two centimetres is four standard deviations for one loaf and twelve and a half standard errors for an average of ten. The test is about the average, so the narrow curve is the one that counts.

The wide orange curve is single loaves, standard deviation 0.5. For one loaf, 17 cm is four standard deviations out: rare, about 3 loaves in 100,000, but loaves like that exist.

\[ \text{one loaf: } z = \frac{17 - 15}{0.5} = 4, \qquad \text{mean of ten: } z = \frac{17 - 15}{0.1581} = 12.65 \]

The narrow blue curve is means of ten loaves. Averaging cancels out the high and low loaves, so the means huddle within half a centimetre of 15. For an average of ten to reach 17, every loaf would have to be tall at once.

This is also where the book's wording slips. It says almost 0 percent of all loaves of bread would be at least as high as 17 cm. The calculation is about means of samples of ten loaves, not about single loaves. The precise sentence is: almost no sample of ten loaves would average 17 cm or more if the population mean were 15.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9's interpretation, restated for sample means

22. The deck's own variation: the baker's mean is 15.3 cm

Worked example

A p-value of essentially zero makes the logic look trivial. Keep everything in Example 9.9 the same, but suppose the ten loaves had averaged 15.3 cm. Find the p-value.

Only the boundary of the tail moves. The null curve is still N(15, 0.1581), because the curve comes from the null and the sample size, not from the sample's mean.

Write the tail

Why: same direction, new boundary.

\[ p\text{-value} = P(\bar{X} \ge 15.3 \mid \mu = 15) \]

Standardise

Why: the distance is now 0.3 cm.

\[ z = \frac{15.3 - 15}{0.1581} = \frac{0.3}{0.1581} = 1.897 \]

Read the right tail

Why: from the standard normal table.

\[ P(Z \ge 1.897) = 1 - 0.9711 = 0.0289 \]

Figure (svg): The same null curve for means of 10 loaves, centred at 15 with standard error 0.158, on an axis from 14.45 to 15.55. A red dashed line marks a sample mean of 15.3, z equals 1.90, and the visible right tail beyond it is shaded red and labelled p equals 0.0289.

Move the sample mean in from 17 to 15.3 and the tail becomes something you can see: about 3 samples in 100 would rise this high by chance alone.

In context: if the loaves really averaged 15 cm, about 3 samples of ten in every 100 would average 15.3 cm or more. That is unusual, but not wildly so, and whether it counts as enough evidence depends on the standard you set before looking, which is Part 3's subject.

Compare this with Example 9.9. The curve did not move, the sample size did not change, and the standard error is the same 0.1581. Only the distance from 15 shrank, from 2 cm to 0.3 cm, and the p-value went from essentially zero to about 3 in 100.

Check with the 68-95-99.7 rule

Why: 1.897 standard errors is just short of 2, where the one-sided tail is 0.025, so the answer should be a little above 0.025.

\[ \tfrac{1}{2}(1 - 0.95) = 0.025 < 0.0289 < \tfrac{1}{2}(1 - 0.68) = 0.16 \]

23. Try It 9.9: a mean of 12.5 from 36 values

Worked example

A normal distribution has standard deviation 1. You want to verify a claim that the mean is greater than 12. A sample of 36 has mean 12.5. The hypotheses are given. Find the p-value and draw it.

\[ H_0: \mu \le 12, \qquad H_a: \mu > 12 \]

The same four moves as the baker: the standard error, the z-score, the tail, the picture. This time the answer is small but not invisible, so the picture earns its keep.

Standard error

Why: sigma over the square root of 36.

\[ \frac{\sigma}{\sqrt{n}} = \frac{1}{\sqrt{36}} = \frac{1}{6} = 0.1667 \]

Standardise the sample mean

Why: half a unit above 12, in sixths.

\[ z = \frac{12.5 - 12}{1/6} = 0.5 \times 6 = 3 \]

The right tail beyond z = 3

Why: the alternative points right.

\[ p\text{-value} = P(Z \ge 3) = 1 - 0.99865 = 0.00135 \]

Figure (svg): A bell curve centred at 12 with standard error one sixth, axis from 11.4 to 12.7 with a tick every standard error. A red dashed line marks the sample mean 12.5, three standard errors above the centre. The sliver of area to its right is shaded red and labelled p equals 0.00135.

Three standard errors out: the red sliver is visible this time, but only just. About 13 samples in 10,000 would land that far right if the mean really were 12.

In context: if the true mean were 12, only about 13 or 14 samples of 36 in every 10,000 would have a mean of 12.5 or more. The book prints 0.0013; the four-decimal value is 0.0013 and the exact tail is 0.00135, which matters on a later slide when the level is 0.001.

Look at the tick marks on the figure: one every sixth of a unit, each a standard error. Counting ticks from 12 to 12.5 gives three, which is the z-score read straight off the picture. Marking the axis in standard errors is the fastest way to sketch any p-value.

Check with the 68-95-99.7 rule

Why: 99.7 percent lies within 3 standard errors, so 0.3 percent lies outside, half of it in each tail.

\[ \tfrac{1}{2}(1 - 0.997) = 0.0015 \approx 0.00135 \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Try It 9.9

24. Slide the sample mean and watch the p-value

Tweak it

Here is Try It 9.9's p-value as a function of the sample mean, for a right-tailed test of a mean of 12 with sigma equal to 1. The curve gives the p-value you would get for each possible x-bar. Change n to see what a bigger sample does.

Parameter explorer

With n at 36, read the p-value at sample means of 12.2, 12.3 and 12.5. Then drop n to 9 and read the p-value at 12.5 again. What does a smaller sample do to the evidence from the same sample mean?

\[ p\text{-value} = P\left(Z \ge (\bar{x} - 12)\sqrt{n}\right), \quad n = {n} \]

  • n — from 4 to 100: sample size n

What to notice: at x-bar equal to 12 the curve sits at exactly 0.5 whatever n is. A sample mean right on the null value is the most ordinary result possible, and half of all samples would beat it.

As x-bar moves right the p-value falls, and it falls faster for a big n. With 100 values, 12.5 is five standard errors out and the p-value is about 3 in 10 million. With 9 values the same 12.5 is only 1.5 standard errors out and the p-value is 0.0668. Evidence depends on the sample mean and on how precisely it was measured.

25. Rank four samples by the strength of evidence

Ranking

All four samples test the same claim as Try It 9.9: a normal population with sigma equal to 1, the null mean at most 12, the alternative greater than 12. They differ in sample mean and in sample size.

Strength of evidence against the null means a smaller p-value. Work out each standard error and z-score before you order them; the sample size changes the answer.

Put in order

Order the samples from the strongest evidence against the null to the weakest.

  1. n = 36, x̄ = 12.5
  2. n = 36, x̄ = 12.3
  3. n = 9, x̄ = 12.5
  4. n = 36, x̄ = 12.2

Why: With 36 values the standard error is one sixth, so 12.5, 12.3 and 12.2 give z of 3, 1.8 and 1.2 and p-values of 0.00135, 0.0359 and 0.1151. With 9 values the standard error is one third, so 12.5 gives z of 1.5 and a p-value of 0.0668, which falls between the 12.3 and 12.2 samples.

\[ n = 9: \quad z = \frac{12.5 - 12}{1/3} = 1.5, \quad p = 0.0668 \]

The small sample with the large mean is the one that catches people. Its mean is the highest, tied with the strongest sample, yet its evidence is weaker than a sample of 36 averaging only 12.3. A small sample has a wide null curve, so even a big departure is not very surprising.

26. The alternative decides which tail

Concept

As extreme or more extreme needs a direction, and the alternative hypothesis supplies it. The baker's alternative said greater than, so the extreme samples were the large ones and the p-value was a right tail.

Figure (svg): Three bell curves side by side, each centred at mu-nought. Left panel, alternative mu less than mu-nought: the left tail beyond the sample mean is shaded red. Middle panel, alternative mu greater than mu-nought: the right tail is shaded. Right panel, alternative mu not equal to mu-nought: both tails are shaded, the mirror image of the observed distance included.

More extreme means further in the direction the alternative points. A two-sided alternative counts a surprise on either side, so its p-value collects both tails.

If the alternative says less than, small sample means are the extreme ones, and the p-value is the left tail. If it says not equal to, a surprise on either side counts, so the p-value collects both tails: the one beyond your sample and its mirror image on the other side.

\[ \text{two-tailed: } p\text{-value} = 2 \times P(Z \ge |z|) \]

Section 9.5 practises all three shapes with full examples. For now, fix one habit: read the sign in the alternative before you shade anything. The sample tells you where the boundary of the tail is; only the alternative tells you which side of it to shade.

For the same z-score a two-tailed p-value is twice the one-tailed one. Try It 9.9's z of 3 would give 0.0027 against a not-equal alternative, instead of 0.00135. The data are the same; the question asked of them is broader.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — left-, right- and two-tailed tests

27. Trap: the probability of exactly the sample you got

Trap

The trap

For the baker, a first attempt computes the chance of the observed mean itself:

\[ p = P(\bar{X} = 17) \]

\[ = 0 \]

It looks precise, but no sample could ever count against the null this way.

Wrong question. It is zero whatever the truth is.

The fix

A sample mean is continuous, so any one exact value has probability zero under every hypothesis. The p-value is the area of the tail beyond your value.

\[ p = P(\bar{X} \ge 17 \mid \mu = 15) \]

For Example 9.9 both happen to come out near zero, which hides the mistake. For Try It 9.9 the exact-value version still gives zero, while the tail gives 0.00135, and for a sample mean of 12.2 the tail is 0.1151, nowhere near zero.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the p-value is the same or greater than 17

28. Two standard errors out: estimate the p-value

Estimation

A right-tailed test of a mean gives a sample mean exactly two standard errors above the null value. You have no table. Estimate.

Predict first

Roughly what is the p-value?

  • About 0.5
  • About 0.16
  • About 0.025
  • About 0.0001

Correct: About 0.025

Why: About 95 percent of a normal curve lies within two standard deviations of its centre, leaving 5 percent outside, half of it in each tail. A right-tailed test counts one tail, so the p-value is about 0.025; the exact value is 0.0228.

\[ P(Z \ge 2) = 1 - 0.9772 = 0.0228 \approx \tfrac{1}{2}(1 - 0.95) \]

Keep three anchors from the empirical rule. One standard error out gives a one-tailed p-value near 0.16; two give about 0.025; three give about 0.0015. Any p-value you compute should agree roughly with these, and if it does not, check the direction of the tail and the standard error.

The anchor 0.16 is worth noticing too. A sample mean a whole standard error above the null value sounds like a departure, yet one sample in six would do that by chance alone.

29. The decision rule

Section

Part 3

30. Compare the p-value with a preset alpha

Concept

A p-value is a measurement. A decision needs a standard to measure it against. The book's systematic way is to compare the p-value with a preset or preconceived alpha, also called the significance level.

significance level α — The probability of a Type I error, rejecting the null when it is true, chosen before the data are collected. It may or may not be given in the problem; if none is given, 0.05 is a common standard.

\[ \alpha > p\text{-value} \;\Rightarrow\; \text{reject } H_0 \qquad \alpha \le p\text{-value} \;\Rightarrow\; \text{do not reject } H_0 \]

Figure (svg): A number line of p-values from 0 to 0.13. A yellow dashed line at 0.05 marks alpha, and the region to its left is tinted red and labelled reject H0 at 0.05. A pink dotted line marks 0.01. Four dots: 0.00135 and 0.025 and 0.0304 lie left of 0.05 in red; 0.1151 lies right of it in blue, labelled do not reject.

The rule is a fence. Any p-value to the left of alpha rejects; any at or to the right does not. Move the fence to 0.01 and the dot at 0.025 changes sides.

When you reject, the results of the sample data are called significant: there is sufficient evidence to conclude that the null is an incorrect belief and that the alternative may be correct. When you do not reject, the results are not significant: there is not sufficient evidence for the alternative.

Read the equality case carefully. The book puts a p-value exactly equal to alpha on the do-not-reject side. To reject, the p-value must be strictly smaller than alpha.

Look at the figure with both fences. At alpha equal to 0.05, three of the four p-values reject. At alpha equal to 0.01, only the smallest does: the dots at 0.025 and 0.0304 cross to the do-not-reject side. Nothing about the data changed; only the standard did.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Decision and Conclusion

31. Reading the decision rule

Notation

The rule is short enough to memorise and easy to turn upside down. Decode it piece by piece.

Annotate

On: \( \alpha > p\text{-value} \;\Rightarrow\; \text{reject } H_0 \)

  • The level you chose before seeing data: how much Type I risk you accept. Common values are 0.10, 0.05 and 0.01.
  • Strictly greater. A p-value equal to alpha does not reject; the book assigns α ≤ p-value to do not reject.
  • Computed from the sample, under the null. The smaller it is, the stronger the evidence against H₀.
  • Means the data are hard to reconcile with H₀. It does not mean H₀ is proved false, or Hₐ proved true.

The book's memory device says the same thing in a rhyme: if the p-value is low, the null must go; if the p-value is high, the null must fly. Low and high here mean below or above alpha, not small or large in any absolute sense.

A p-value of 0.03 is low against an alpha of 0.05 and high against an alpha of 0.01. The rhyme only works once you have said which alpha you are comparing with.

One more caution about direction. Memorise the rule as p greater than alpha means reject and you will get every decision backwards. Anchor it to meaning instead: a smaller p-value is stronger evidence, so rejection belongs to the small p-values.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Example 9.10's memory device

32. Example 9.9 decided: the baker at the 5 percent level

Worked example

The book stops Example 9.9 at the p-value, which is essentially zero. No significance level is given. Make the decision and say what it means.

Section 9.5 gives the default to use when no level is stated: alpha equal to 0.05. Write it down before comparing, so the reader knows the standard that was applied.

Fix the level

Why: none is given, so use the common standard and say so.

\[ \alpha = 0.05 \]

Recall the p-value

Why: from the previous calculation.

\[ p\text{-value} \approx 5.7 \times 10^{-37} \]

Compare

Why: alpha is far larger than the p-value.

\[ 0.05 > 5.7 \times 10^{-37} \]

Apply the rule

Why: alpha greater than the p-value means reject.

\[ \text{reject } H_0: \mu \le 15 \]

In context: the results are significant. At the 5 percent level there is sufficient evidence that the population mean height of the baker's loaves is greater than 15 cm. With a p-value this small, the decision would be the same at any level anyone uses.

The remark about any level is worth unpacking. Rejection needs alpha above the p-value, and nobody uses an alpha anywhere near 10 to the minus 37. When a p-value is this extreme the choice of level stops mattering; it matters only for results near the fence.

Check with the cutoff for the sample mean

Why: at the 0.05 level the rejection region starts 1.645 standard errors above 15, and 17 is far beyond it.

\[ 15 + 1.645(0.1581) = 15.26 < 17 \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9's conclusion

33. Example 9.10: fill in the rule

Fill the middle

The book's Example 9.10 asks you to complete four sentences. Fill in the missing words, using comparisons with alpha.

Fill in the blanks

Reject the null hypothesis when the p-value is less than alpha. The results of the sample data then support the alternative hypothesis. Do not reject the null hypothesis when the p-value is greater than or equal to alpha.

Why: A p-value below alpha rejects the null and the sample data support the alternative. A p-value at or above alpha does not reject, and the data do not support the alternative. The book's solution writes greater than alpha for the second case; the rule on page 468 adds the equal case to it.

Two small repairs to the book here. Its blank reads do not reject the null when hypothesis when, a typo for do not reject the null hypothesis when. And its solution says greater than alpha, leaving out a p-value exactly equal to alpha; the decision rule on the previous page puts that case on the do-not-reject side, so the sentence above says greater than or equal to.

Notice the careful verb in the second sentence. The data support the alternative; they do not prove it. Support is the strongest word a single test earns.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Example 9.10

34. Try It 9.10: It's a Boy Genetics Labs

Worked example

It's a Boy Genetics Labs claim their procedures improve the chances of a boy being born. A test of a single population proportion gives the results below. Interpret them and state a conclusion in simple, non-technical terms.

\[ H_0: p = 0.50, \quad H_a: p > 0.50, \quad \alpha = 0.01, \quad p\text{-value} = 0.025 \]

Two different p's appear in that line, which is a nuisance of notation. The p in the hypotheses is the population proportion of boys; the p-value is a probability about samples. Keep them apart before comparing anything.

Name the parameter

Why: p is the proportion of births that are boys under the labs' procedures.

\[ p = \text{proportion of boys born} \]

Compare alpha with the p-value

Why: the level was set at 1 percent.

\[ \alpha = 0.01 < 0.025 = p\text{-value} \]

Apply the rule

Why: alpha is at or below the p-value, so do not reject.

\[ \text{do not reject } H_0 \]

Say what the p-value meant

Why: if the procedures did nothing, about 2.5 percent of samples would show this many boys or more.

\[ P(\text{this many boys or more} \mid p = 0.5) = 0.025 \]

In plain terms: at the 1 percent level, the data do not give enough evidence that the labs' procedures raise the chance of having a boy. The results were somewhat unusual for an ordinary fifty-fifty chance, but not unusual enough to meet the strict standard the test set.

The word simple in the question matters. A parent reading this result does not know what a null hypothesis is, so the conclusion has to talk about babies and boys, not about H0 and alpha.

Check what the conclusion does not say

Why: not rejecting is not evidence that the procedures fail; at alpha equal to 0.05 the same p-value would have rejected, which is why the level must be stated.

\[ 0.05 > 0.025 \;\Rightarrow\; \text{reject at the 5 percent level} \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Try It 9.10

35. Try It 9.9 at two levels: 0.01 and 0.001

Worked example

Try It 9.9's p-value was 0.00135. Decide the test at alpha equal to 0.01 and again at alpha equal to 0.001, and draw both rejection regions.

Same data, two standards. The comparison is one line each; the figure shows why the two answers differ.

At the 1 percent level

Why: compare 0.01 with 0.00135.

\[ \alpha = 0.01 > 0.00135 \;\Rightarrow\; \text{reject } H_0 \]

At the 0.1 percent level

Why: compare 0.001 with 0.00135.

\[ \alpha = 0.001 < 0.00135 \;\Rightarrow\; \text{do not reject } H_0 \]

Find each cutoff for the sample mean

Why: the z for a right tail of 0.01 is 2.326; for 0.001 it is 3.090.

\[ 12 + 2.326 \cdot \tfrac{1}{6} = 12.388, \qquad 12 + 3.090 \cdot \tfrac{1}{6} = 12.515 \]

Figure (svg): The Try It 9.9 null curve, centred at 12 with standard error one sixth. The pink region beyond 12.388 is the rejection region for alpha 0.01; the smaller red region beyond 12.515 is the rejection region for alpha 0.001. A yellow line marks the sample mean 12.5, which falls inside the pink region but just short of the red one.

Same data, two fences. The sample mean clears the 0.01 cutoff with room to spare and misses the 0.001 cutoff by fifteen thousandths of a unit.

In context: at the 1 percent level there is sufficient evidence that the mean is greater than 12; at the 0.1 percent level there is not. This is where the book's rounded 0.0013 would mislead you: it is above 0.001 either way, but 0.00135 shows how close the call really is.

The picture shows why both decisions are right at once. Each alpha draws its own rejection region on the null's curve, and the smaller the alpha, the further right the region begins. Comparing the p-value with alpha and comparing the sample mean with the cutoff are the same test seen two ways.

Check the decisions against the cutoffs

Why: the sample mean must lie beyond a cutoff exactly when alpha exceeds the p-value.

\[ 12.5 > 12.388 \;\text{(reject)}, \qquad 12.5 < 12.515 \;\text{(do not reject)} \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Try It 9.9 and the decision rule

36. A stricter alpha: more rejections or fewer?

Prediction

Two researchers analyse the same data. One tests at alpha equal to 0.05, the other at 0.01. Commit to an answer before reasoning it through.

Predict first

Which researcher is more likely to reject the null hypothesis?

  • The one using 0.05
  • The one using 0.01
  • They always reach the same decision
  • It depends only on the sample size

Correct: The one using 0.05

Why: Rejection needs the p-value to fall below alpha. Every p-value below 0.01 is also below 0.05, but p-values between 0.01 and 0.05 reject only for the researcher using 0.05. A smaller alpha is a stricter standard and rejects less often.

People often mix up which way stricter goes. A small alpha means a small chance of a false alarm, so it demands rarer data before rejecting. The price is that real effects are missed more often: the Type II error from Section 9.2 becomes more likely.

So choosing alpha is choosing a balance between the two errors. It has to be done before the data are seen, for a reason a simulation a few slides on makes visible.

37. Reject, or do not reject?

Sorting

Apply the decision rule to each pair. Compare alpha with the p-value, and remember which side the equal case falls on.

Sort into buckets

Sort each test result by its decision.

reject H₀: α > p-value
p-value = 0.00135, α = 0.01; p-value = 0.0304, α = 0.05; p-value = 0.0007, α = 0.001; p-value = 0.0289, α = 0.10
do not reject H₀: α ≤ p-value
p-value = 0.025, α = 0.01; p-value = 0.1151, α = 0.05; p-value = 0.0359, α = 0.025; p-value = 0.05, α = 0.05
rej
In each, alpha is strictly larger than the p-value: the sample would be rarer under the null than the standard allows.
no
In each, the p-value is at least alpha. That includes the pair where the two are exactly equal, which the book's rule sends to do not reject.

Look at the pair with a p-value of 0.0359. It would reject at 0.05, the default, but this test was set at 0.025, so it does not. The decision belongs to a p-value and a level together; neither number alone can make it.

And the 0.05 against 0.05 pair is the one to remember. It is low by any everyday standard, but the rule asks whether alpha is greater than the p-value, and equal is not greater.

38. When the p-value lands exactly on alpha

Edge cases

Push the rule to its edge. With continuous data a p-value is almost never exactly equal to alpha, but with rounded numbers, or a discrete count, it can be.

Discussion prompt

A test at alpha equal to 0.05 reports a p-value of exactly 0.0500. What is the decision, and why does the book draw the line on that side?

The tie rule follows the logic of the whole chapter. The null is the default and the alternative must earn its place. A result that only just matches the standard has not exceeded it.

There is a second lesson in the edge case. Results very close to the fence, whichever side they land, carry almost the same evidence. Part 5 comes back to this: the decision is a yes or no, but the evidence is a sliding scale.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — If α ≤ p-value, do not reject H0

39. Why alpha is fixed before the data arrive

Concept

The book insists that the statistician selects alpha before collecting the sample data. The reason is visible in a simulation. Here are 1,000 right-tailed tests in which the null hypothesis is true every time; each gave a p-value.

Figure (svg): A histogram of 1,000 p-values from simulated right-tailed tests in which the null hypothesis is true. Twenty bins each 0.05 wide from 0 to 1, all roughly the same height near 50. The first bin, p-values below 0.05, is red and holds 39. A dashed line marks the expected height 50.

When the null is true, a p-value is equally likely to land anywhere from 0 to 1. So a fence set at 0.05 in advance is crossed by about 5 percent of true nulls, and no more.

The p-values are spread evenly from 0 to 1. That is a general fact: when the null is true, the p-value is equally likely to fall anywhere. So the fraction falling below 0.05 is about 0.05. In this run 39 of the 1,000 did, close to the 50 expected.

\[ P(p\text{-value} < \alpha \mid H_0 \text{ true}) = \alpha \]

That line is the promise behind alpha: the probability of a Type I error is exactly the level you set. Now break the promise. Suppose you look at the p-value first and then pick a level that makes it significant, using 0.10 whenever the p-value lands between 0.05 and 0.10. In this run 82 true nulls would be rejected, not 39.

Choosing alpha after seeing the data quietly doubles the false-alarm rate in that example, and nothing in the final report reveals it. Fixing alpha first is what makes the significance level mean what it says.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — the statistician selects α before collecting the sample data

40. Who sets alpha, and how strict?

Real world

If no level of significance is given, the book says a common standard is alpha equal to 0.05, and you should state it in the conclusion. In practice, the field and the cost of a false alarm set the level.

Particle physicists announce a discovery only at five standard errors, a one-tailed p-value of about 3 in 10 million. Medical trials commonly use 0.05, sometimes 0.01 for a confirmatory trial. A market researcher deciding which of two adverts to run might happily use 0.10.

\[ P(Z \ge 5) \approx 2.9 \times 10^{-7} \]

Discussion prompt

Why would a physicist demand an alpha thousands of times smaller than a market researcher's? Answer in terms of the Type I error from Section 9.2.

Whichever level is chosen, it is chosen in advance and stated in the conclusion. A result reported as significant without its level is incomplete, because significant at 0.10 and significant at 0.001 are very different claims.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — If no level of significance is given, a common standard is α = 0.05

41. What a p-value is not

Section

Part 4

42. Data given the null, not the null given the data

Concept

The p-value is computed by assuming the null is true. That one fact rules out the most common misreading of it: the p-value is not the probability that the null hypothesis is true.

\[ p\text{-value} = P(\text{data this extreme} \mid H_0) \;\ne\; P(H_0 \mid \text{data}) \]

Swapping the two sides of a given-that bar changes the question completely. The probability that a person speaks English, given that they are a United States senator, is essentially 1. The probability that a person is a senator, given that they speak English, is tiny. Same two events, very different numbers.

The left-hand quantity is what a test can compute, because the null hands you a curve. The right-hand quantity would need to know how plausible the null was before you collected any data: how many of the claims tested in situations like this are true. The test has no such information.

A null hypothesis is either true or false; it does not have a probability in this framework. What has a probability is the sample, which could have come out differently. The randomness lives in the data, exactly as it did for confidence intervals in Chapter 8.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — the p-value assumes the null hypothesis is true

43. The deck's own example: a thousand tested hypotheses

Worked example

Imagine 1,000 studies, each testing a null at alpha equal to 0.05. Suppose 900 of the nulls are actually true and 100 are false, and each test has power 0.80 against a false null. Among all the rejections, what fraction are true nulls?

Every rejected study reported a p-value below 0.05. If a p-value were the probability that the null is true, at most 5 percent of these rejections could be wrong. Count them and see.

False alarms among the true nulls

Why: each true null is rejected with probability alpha.

\[ 900 \times 0.05 = 45 \]

Real findings among the false nulls

Why: each false null is caught with probability equal to the power.

\[ 100 \times 0.80 = 80 \]

Total rejections

Why: add the two kinds.

\[ 45 + 80 = 125 \]

Fraction of rejections that were true nulls

Why: false alarms over all rejections.

\[ \frac{45}{125} = 0.36 \]

Figure (svg): A tree. The root box: 1,000 nulls tested at alpha 0.05. It branches into 900 true nulls and 100 false nulls. The 900 true nulls branch into 45 rejected, false alarms, in red, and 855 not rejected. The 100 false nulls branch into 80 rejected, real findings, in green, and 20 missed. A bracket gathers the two rejected boxes: 125 rejections, of which 45, that is 36 percent, are true nulls.

Every one of the 125 rejections came with a p-value below 0.05. Over a third of them are wrong anyway, and nothing in any single p-value warns you which.

In context: 36 percent of the significant results in this field would be false alarms, even though every one passed the 5 percent standard. The figure depends on how many of the tested nulls were true to begin with, which is exactly the information a single p-value does not contain.

Change the field and the answer changes. If half the tested nulls were false instead of a tenth, the false alarms would fall to 25 out of 425 rejections, about 6 percent. The p-values could be identical in both fields; what differs is how plausible the nulls were before anyone looked.

Check that the tree accounts for every study

Why: the four leaves must add back to 1,000.

\[ 45 + 855 + 80 + 20 = 1000 \]

44. Trap: the p-value as the chance the null is true

Trap

The trap

A report says the p-value was 0.03 and concludes:

\[ P(H_0 \text{ true}) = 0.03 \]

so there is a 97 percent chance the effect is real.

It reads naturally, which is why it turns up so often in news reports.

Wrong. The 0.03 was computed assuming the null is true.

The fix

The p-value is a probability about the data, given the null:

\[ P(\text{data this extreme} \mid H_0) = 0.03 \]

Correct reading: if the null were true, a sample this extreme would turn up about 3 times in 100. How likely the null itself is was never computed, and the tree on the previous slide shows it can be far larger than 0.03.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 467 — the definition of the p-value

45. Four readings of a p-value of 0.03 at alpha 0.05

Error analysis

A test at alpha equal to 0.05 returns a p-value of 0.03. Four people write it up. Only one of them is right; hover over each to see what goes wrong in the others.

Annotate

On: \( \begin{aligned} &(1)\; \text{do not reject, since } 0.03 < 0.05 \\ &(2)\; \text{reject; a 3 percent chance } H_0 \text{ is true} \\ &(3)\; \text{reject; so the effect is large} \\ &(4)\; \text{reject; data this extreme: 3 in 100 if } H_0 \end{aligned} \)

  • The rule turned upside down. Alpha 0.05 is greater than the p-value 0.03, which means reject.
  • Right decision, wrong meaning. The 3 percent is computed assuming H₀ is true, so it cannot be the chance H₀ is true.
  • Right decision, wrong meaning. A small p-value can come from a tiny effect measured on a huge sample.
  • Correct: the decision and the meaning, with the condition stated.

Readings two and three are the dangerous ones, precisely because they reach the right decision. Nothing in the conclusion flags the faulty reasoning, so they survive into reports and news stories. Only the explanation exposes them.

The habit that protects you is to say the condition out loud every time: if the null were true, data this extreme would happen this often. A sentence about a p-value that leaves out the if has almost always gone wrong.

46. The deck's own example: significant is not the same as large

Worked example

A test score has standard deviation 15. A new study method raises the mean score by just 0.1 point, an effect nobody would notice. Find the right-tailed p-value for that difference with a sample of 100 and with a sample of 100,000.

The difference is the same in both studies; only the sample size changes. Watch what the sample size does to the standard error, and through it to the z-score.

Standard error with 100

Why: 15 over the square root of 100.

\[ \frac{15}{\sqrt{100}} = \frac{15}{10} = 1.5 \]

z and p-value with 100

Why: a tenth of a point is a tiny fraction of 1.5.

\[ z = \frac{0.1}{1.5} = 0.067, \quad p = P(Z \ge 0.067) = 0.473 \]

Standard error with 100,000

Why: 15 over the square root of 100,000.

\[ \frac{15}{\sqrt{100000}} = \frac{15}{316.23} = 0.0474 \]

z and p-value with 100,000

Why: now a tenth of a point is two standard errors.

\[ z = \frac{0.1}{0.0474} = 2.108, \quad p = P(Z \ge 2.108) = 0.0175 \]

Figure (svg): Two bell curves, both the null's sampling distribution of a mean difference centred at zero. Left, n equals 100, standard error 1.5, axis from minus 5 to 5: the observed difference 0.1 sits almost at the centre, and nearly half the curve, 0.47, is shaded beyond it. Right, n equals 100,000, standard error 0.047, axis from minus 0.2 to 0.2: the same 0.1 sits out in the tail, and only a red sliver, 0.0175, lies beyond it.

The difference is a tenth of a point in both panels. Only the width of the curve changed, and the width is set by the sample size, not by how much the difference matters.

In context: the large study declares a significant improvement at the 5 percent level, and the improvement is a tenth of a point on a test with a standard deviation of 15. Significant means hard to explain by chance; it does not mean big, and it does not mean important.

Carry this into reading research. When a study reports a significant result, look for the size of the effect before deciding whether it matters. A huge sample makes almost any difference significant, so significance alone tells you little about how much changed.

Check the ratio of the standard errors

Why: a sample 1,000 times larger shrinks the standard error by the square root of 1,000.

\[ \frac{1.5}{0.0474} = 31.6 = \sqrt{1000} \]

47. An important effect that is not significant

Counterexample

The previous slide showed significance without importance. The reverse also happens, and it is just as easy to misread.

A small pilot study of a reading programme tests 9 pupils. The mean score rises by 8 points on a test with standard deviation 15, a large and educationally important gain. The test is right-tailed at alpha equal to 0.05.

\[ \frac{15}{\sqrt{9}} = 5, \quad z = \frac{8}{5} = 1.6, \quad p = P(Z \ge 1.6) = 0.0548 \]

Discussion prompt

The p-value is 0.0548, so the test does not reject at the 5 percent level. Does this show the programme does not work? What should the researchers say instead?

Put the two slides together and the p-value's job comes into focus. It measures how hard the data are to explain by chance, which depends on the size of the effect and on the size of the sample at once. To know how big an effect is, look at the effect itself: the 8 points, or the 0.1 point.

48. Trap: a large p-value read as proof of the null

Trap

The trap

The tutoring pilot gives a p-value of 0.0548 and the report says:

\[ p > 0.05 \]

\[ \Rightarrow \text{ the programme has no effect} \]

The arithmetic and the decision are both right; the last line claims far more than they support.

Wrong. Not rejecting is not accepting.

The fix

The book is explicit: when you do not reject, it does not mean you should believe the null is true. The data have simply failed to provide sufficient evidence against it.

\[ \text{do not reject } H_0 \]

\[ = \text{ not sufficient evidence for } H_a \]

A small sample can fail to detect a large effect. Absence of evidence, from 9 pupils, is not evidence of absence.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — the third bullet of the decision rule

49. Correct reading, or misreading?

Discrimination

A right-tailed test of whether a new fertiliser increases mean yield gives a p-value of 0.02. Sort each statement. The test for every one: does it keep the condition that the null is true, and does it claim only what a p-value can say?

Sort into buckets

Sort each statement about the p-value of 0.02.

a correct reading
if the fertiliser did nothing, a yield increase this large or larger would occur in about 2% of samples; the sample would be unusual if the null were true; at α = 0.05 the result is statistically significant
a misreading
there is a 2% chance that the fertiliser does nothing; there is a 98% chance the fertiliser works; the fertiliser must produce a large increase in yield
ok
Each keeps the p-value as a statement about samples under the null, or applies the decision rule correctly: 0.05 is greater than 0.02.
bad
Two flip the condition, turning a probability about data into one about the hypothesis. The third confuses small p-value with large effect.

The statement about a 98 percent chance deserves a second look, because it sounds modest and careful. It is the same error as the 2 percent chance the fertiliser does nothing, just written as a complement. Neither number exists in the test.

A correct sentence about a p-value has a recognisable shape: it starts with if the null were true, or ends with under the null. When you write one yourself, check for that clause.

50. Each misreading, and the fact that breaks it

Matching

Every common misreading of a p-value is broken by one specific fact about how it was computed. Match them.

Match the pairs

  • a. p = 0.03 means a 3% chance H₀ is true
  • b. p = 0.40 shows H₀ is true
  • c. p = 0.001 means the effect is large
  • d. p = 0.04 at α = 0.05 is about as strong as p = 0.001
  • w. the p-value is computed assuming H₀ is true
  • x. ordinary data fit H₀ and its neighbours equally well
  • y. a huge sample makes even a tiny effect many SEs out
  • z. the decision is yes or no, but the evidence is a scale

Why: A probability computed by assuming the null cannot measure the null. A large p-value says the data are ordinary under the null, which does not single the null out. Smallness of p mixes effect size with sample size. And two p-values on the same side of alpha can differ in strength by a factor of forty.

These four facts are worth more than a list of forbidden sentences. A new misreading you have never seen before will almost always break against one of them.

The last pair leads into Part 5. The decision rule sorts results into two bins, but the book itself warns against stopping there: a data analyst should have more confidence in a rejection at 0.001 than one at 0.04, even with alpha at 0.05.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — Thinking about the meaning of the p-value

51. Writing the conclusion

Section

Part 5

52. A conclusion has four parts

Concept

After you make your decision, the book asks you to write a thoughtful conclusion about the hypotheses in terms of the given problem. Reject H0 is not a conclusion; it is a step. The conclusion is a sentence someone who never saw your arithmetic could act on.

  1. The level: at the 5 percent level, or at α = 0.01
  2. The evidence: there is sufficient evidence, or there is not sufficient evidence
  3. The claim: in the alternative's direction, in the problem's own words
  4. The population: the parameter as a real quantity, with units

\[ \text{At the } \alpha \text{ level, there is (not) sufficient evidence that } \langle H_a \text{ in words} \rangle \]

The template is always about the alternative. When you reject, the evidence is sufficient for it; when you do not reject, the evidence is not sufficient for it. You never conclude the null is true, and you never say the alternative is proved.

Notice that the conclusion is about the population. The sample mean of the baker's ten loaves was 17 cm; that is a fact, and needs no test. The claim the test speaks to is about all the loaves he bakes.

The word not is the only thing that changes between the two decisions. The level, the word evidence and the claim stay fixed, which makes the conclusion almost mechanical once the decision is made.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 469 — Conclusion: write a thoughtful conclusion in terms of the given problem

53. Example 9.9 concluded: the baker's sentence

Worked example

The baker's test rejected the null at the 5 percent level. Build the conclusion piece by piece, then check it would make sense to one of his doubtful customers.

Each step adds one part of the template from the previous slide. Write them in order and the sentence assembles itself.

The level

Why: the standard that was applied, fixed before the data.

\[ \text{At the 5 percent significance level,} \]

The evidence

Why: the test rejected, so the evidence was sufficient.

\[ \text{there is sufficient evidence} \]

The claim, in the alternative's direction

Why: greater than 15.

\[ \text{that the mean height is greater than } 15 \text{ cm} \]

The population

Why: all the loaves he bakes, not the ten in the sample.

\[ \text{for the population of the baker's loaves.} \]

Put together, this is almost word for word the book's own conclusion: there is sufficient evidence that the true mean height for the population of the baker's loaves of bread is greater than 15 cm.

Now read it as one of the doubtful customers would. It tells them what standard was used, what the evidence showed and what it was about, with no z-score in sight. That is the test of a good conclusion: someone who never saw the working could act on it.

Check the sentence for forbidden words

Why: it says sufficient evidence, not proved; it is about the population mean, not the sample; and it names the level.

\[ \text{no proof, no acceptance, no missing level} \]

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, p. 468 — Example 9.9's conclusion

54. The deck's own example: is the coffee machine under-filling?

Worked example

A café's machine is set to fill cups with 8 ounces on average; fills are normal with sigma equal to 0.4 ounce. The owner suspects it is under-filling. A random sample of 25 cups averages 7.85 ounces. Test at the 5 percent level and write the conclusion.

A full test, start to finish. The suspicion is that the mean is less than 8, so the alternative points left and so will the tail.

Hypotheses

Why: under-filling means less than 8; the null takes the complement.

\[ H_0: \mu \ge 8, \qquad H_a: \mu < 8 \]

Standard error

Why: sigma is known and fills are normal: a z-test.

\[ \frac{0.4}{\sqrt{25}} = \frac{0.4}{5} = 0.08 \]

Standardise

Why: the sample sits below the null value.

\[ z = \frac{7.85 - 8}{0.08} = \frac{-0.15}{0.08} = -1.875 \]

The left tail

Why: the alternative points left.

\[ p\text{-value} = P(Z \le -1.875) = 0.0304 \]

Figure (svg): A bell curve centred at 8 ounces with standard error 0.08, axis from 7.7 to 8.3 with ticks every standard error. A red dashed line marks the sample mean 7.85, labelled z equals minus 1.875, and the left tail below it is shaded red and labelled p equals 0.0304.

The alternative points left, so smaller means are the extreme ones and the p-value is the left tail. About 3 in 100 samples of 25 cups would be this light from a machine that really averages 8 ounces.

Decide

Why: alpha is greater than the p-value.

\[ 0.05 > 0.0304 \;\Rightarrow\; \text{reject } H_0 \]

Conclusion: at the 5 percent level there is sufficient evidence that the machine's mean fill is less than 8 ounces. Note what it does not say: it does not say the machine fills 7.85 ounces, only that the 8-ounce setting is no longer believable.

At the 1 percent level the same p-value of 0.0304 would not reject. So before calling the engineer, the owner should have decided which mistake is worse: adjusting a machine that was fine, or leaving a short-filling machine in service.

Check with the cutoff

Why: at 0.05 a left-tailed test rejects below 8 minus 1.645 standard errors, and 7.85 is below it.

\[ 8 - 1.645(0.08) = 7.868 > 7.85 \]

55. The deck's own example: a claim the data do not support

Worked example

A tutoring centre claims that more than half of its students raise their grade within a term. In a random sample of 100 students, 56 did. Test the claim at the 5 percent level and write the conclusion.

A test of a proportion, from Section 9.3's third row. The conditions need checking first, then the same four moves as a mean.

Hypotheses

Why: the claim is more than half, so it goes in the alternative.

\[ H_0: p \le 0.50, \qquad H_a: p > 0.50 \]

Check the normal approximation

Why: both np and nq must exceed five.

\[ np = 100(0.5) = 50 > 5, \quad nq = 50 > 5 \]

Standard error under the null

Why: use the null's p, not the sample's.

\[ \sqrt{\frac{pq}{n}} = \sqrt{\frac{0.5 \times 0.5}{100}} = \sqrt{0.0025} = 0.05 \]

Standardise and take the right tail

Why: the sample proportion is 0.56.

\[ z = \frac{0.56 - 0.50}{0.05} = 1.2, \quad p\text{-value} = P(Z \ge 1.2) = 0.1151 \]

Figure (svg): A bell curve centred at 0.50 with standard error 0.05, axis from 0.33 to 0.67. A yellow dashed line marks the sample proportion 0.56, z equals 1.2, and the right tail beyond it is shaded blue and labelled p equals 0.1151, a visibly substantial area.

This tail is easy to see: more than one sample in nine would show 56 or more improvers even if the true rate were exactly one half. That is not a rare event.

Decide

Why: alpha is at or below the p-value.

\[ 0.05 < 0.1151 \;\Rightarrow\; \text{do not reject } H_0 \]

Conclusion: at the 5 percent level there is not sufficient evidence that more than half of the centre's students raise their grade within a term. It does not say that half or fewer do; 56 out of 100 is simply not unusual enough for a fifty-fifty rate to be doubted.

Notice the standard error used the null's proportion, 0.5, not the sample's 0.56. The curve describes the imaginary world in which the null is true, so every number that builds it comes from the null; the sample only supplies the boundary of the tail.

Check the p-value against the empirical rule

Why: 1.2 standard errors is a little past 1, where the one-sided tail is about 0.16, so the answer should be a little below 0.16.

\[ 0.1151 < 0.16 \]

56. How much the fence throws away

Picture it

The book's Section 9.5 adds a note about the meaning of the p-value that is easy to skip and worth keeping. Here are four p-values against a fence at 0.05, on a scale where each step is ten times smaller.

Figure (svg): A logarithmic number line of p-values from 0.0005 to 1, with a yellow dashed fence at alpha 0.05. Red dots at 0.001 and 0.04 lie left of the fence and are bracketed together as same decision, 40 times apart. Blue dots at 0.056 and 0.4 lie right of it. A pink bracket joins 0.04 and 0.056 across the fence: nearly the same evidence, opposite decisions.

The fence turns a sliding scale into two bins. The p-value keeps what the bins throw away, which is why the book asks for judgement and not a mindless rule.

The red pair, 0.001 and 0.04, land on the same side of the fence, yet one is forty times smaller than the other. The book says an analyst should have more confidence in the rejection at 0.001.

The pink pair, 0.04 and 0.056, land on opposite sides, yet they are almost the same evidence. Likewise a p-value of 0.4 gives much more confidence in not rejecting than 0.056 does. This, in the book's words, makes the analyst use judgement rather than mindlessly applying rules.

So report both: the decision at the preset level, which is what the procedure promised, and the p-value itself, which tells the reader how close the call was. A reader with a stricter or looser standard than yours can then make their own decision from your report.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.5, p. 469 — Thinking about the meaning of the p-value

57. Is this conclusion acceptable?

Sorting

Every sentence below reports the coffee-machine test, which rejected the null at the 5 percent level with a p-value of 0.0304. Sort them. An acceptable sentence names the level, speaks of evidence and describes the population mean in context.

Sort into buckets

Sort each sentence about the coffee-machine test.

acceptable conclusion
At the 5% level there is sufficient evidence that the machine's mean fill is less than 8 ounces.; The data support the claim that the mean fill is below 8 ounces (α = 0.05, p = 0.0304).
not acceptable
We have proved the machine under-fills.; Reject H₀.; The machine's mean fill is 7.85 ounces.; There is a 3% chance the machine is filling correctly.
ok
Both state the level, speak of evidence or support rather than proof, and describe the population mean in the problem's words.
bad
One claims proof, one is a step with no context, one reports the sample mean as if it were the population's, and one turns the p-value into the chance the null is true.

The sentence that gives 7.85 ounces as the machine's mean is worth a pause. That is the sample's mean, a fact about 25 cups. The test makes no claim about where the population mean is, only that 8 is no longer believable.

And the bare Reject H0 is not wrong, merely unfinished. It is the decision, the fifth box in the procedure; the conclusion is the sixth.

58. Before you write a word: whose words?

Step zero

A conclusion has to use the problem's own words, so before computing anything, collect them. Here is a new claim to practise on.

A city says its buses arrive late on fewer than 10 percent of trips. A rider's group samples 400 trips at random and finds 52 late ones. The group will test at alpha equal to 0.01.

Discussion prompt

Before any arithmetic: what is the parameter in words, what does the alternative say in words, what level will the conclusion name, and which tail will the p-value be?

Doing this first protects the conclusion from the most common failure: a correct calculation reported in words that do not match the question. The sample proportion here is 52 over 400, which is 0.13, but that number belongs in the working, not in the claim.

If you carry the test through, the standard error is 0.015, z is 2, and the p-value is 0.0228, so at the 1 percent level the riders do not have sufficient evidence, even though they would at 5 percent.

59. Five tests, decided and concluded

Comparison

Each row is a test from this deck. Fill in the decision and whether the evidence is sufficient for the alternative.

Comparison matrix

testHₐαp-valuedecisionevidence for Hₐ
bakerμ > 15 cm0.05≈ 0rejectsufficient
Try It 9.9μ > 120.0010.00135do not rejectnot sufficient
Genetics Labsp > 0.500.010.025do not rejectnot sufficient
coffee machineμ < 8 oz0.050.0304rejectsufficient
tutoring centrep > 0.500.050.1151do not rejectnot sufficient

Read down the decision column against the last two numbers. Every reject has alpha strictly greater than the p-value; every do-not-reject has alpha at or below it. The evidence column then follows the decision automatically.

Now read across the Try It 9.9 row. Its p-value is the second smallest in the table, yet it is a do-not-reject, because its alpha is the strictest. A p-value is only small or large relative to a level.

60. Putting it together

Section

Part 6

61. Pattern: from a claim to a conclusion

Pattern

Every test in this deck ran through the same six steps. The arithmetic lives in steps 3 and 4; the marks are usually lost in steps 1, 5 and 6.

Figure (svg): A flow of six boxes. 1 hypotheses, from the claim. 2 assume H0: draw its sampling distribution. 3 locate the sample: z equals observed minus null over standard error. 4 p-value: the tail the alternative points to. 5 decide: alpha greater than p, reject; otherwise do not. 6 conclude: level, evidence, claim in context.

Read it as a loop that ends where it began: the conclusion is written in the words of the claim from step 1.
  1. Hypotheses from the claim: the direction word fixes the alternative; the null is its complement.
  2. Assume the null at its boundary value and draw its sampling distribution, with the standard error.
  3. Locate the sample: a z-score, the distance from the null value in standard errors.
  4. The p-value is the tail the alternative points to. Sketch it and check it against 0.16, 0.025 and 0.0015.
  5. Decide with the preset alpha: reject only if alpha is strictly greater than the p-value.
  6. Conclude at the stated level: sufficient or not sufficient evidence for the alternative, in context.

Use the list as a checklist on your own work. Each line names a decision, and most wrong answers in this section come from making one of them silently: a tail chosen without reading the alternative, a level chosen after seeing the p-value, a conclusion written about the sample.

\[ p\text{-value} = P(\text{as extreme or more} \mid H_0), \qquad \alpha > p \Rightarrow \text{reject} \]

Two habits hold the pattern together. Fix the hypotheses and alpha before you look at the sample. And keep the condition that the null is true attached to every sentence you write about a p-value.

62. Check: what the p-value means

Check

Pick the sentence that keeps the condition in the right place.

Before looking at the options, say the definition to yourself, including the if clause.

Check your understanding

A test of whether a new battery lasts longer than the old one gives a p-value of 0.01. Which statement is correct?

  • A. If the new battery lasted no longer on average, a sample this extreme or more would occur about 1% of the time. (correct)
  • B. There is a 1% chance that the new battery lasts no longer on average.
  • C. There is a 99% chance that the new battery lasts longer on average.
  • D. The new battery lasts much longer than the old one.

Answer: A

Why: The p-value is computed by assuming the null is true, here that the new battery lasts no longer, and it gives the chance of a sample at least as extreme as the one observed. It says nothing directly about the probability of either hypothesis.

Why B tempts people
That turns the condition round: it gives the chance of the null given the data, which the test never computes.
Why C tempts people
The same reversal written as a complement. No 99 percent figure exists anywhere in the test.
Why D tempts people
A small p-value does not mean a large effect; a big sample can make a small improvement significant.

If you chose correctly, push yourself one step further: rewrite the correct option for a p-value of 0.30. It still begins with the same if clause; only the frequency changes.

Two of the wrong options are the same mistake in different clothes: one gives the probability of the null, the other the probability of the alternative. Both swap the two sides of the given-that bar.

The option about a much longer life confuses significance with size. To know how much longer the new battery lasts, look at the difference in mean lifetimes, not at the p-value.

63. Check: applying the decision rule

Check

Compare, then decide; then make sure the words match the decision.

Write the comparison as an inequality with alpha on the left before reading the options.

Check your understanding

A test is run at the 1 percent level and gives a p-value of 0.018. What is the correct decision and conclusion?

  • A. Do not reject H₀: there is not sufficient evidence for Hₐ at the 1% level. (correct)
  • B. Reject H₀: 0.018 is a small p-value.
  • C. Do not reject H₀: the data show that H₀ is true.
  • D. Reject H₀: there is sufficient evidence for Hₐ at the 1% level.

Answer: A

Why: Alpha, 0.01, is less than the p-value, 0.018, so the rule says do not reject. The conclusion is that the evidence is not sufficient for the alternative at the 1 percent level, nothing stronger.

Why B tempts people
0.018 is small in everyday terms but not below this test's alpha. Small only means something relative to the preset level.
Why C tempts people
The decision is right but the reading is not: failing to reject never shows the null is true.
Why D tempts people
This applies the rule backwards. Rejection needs alpha strictly greater than the p-value.

Read the inequality aloud as a sentence: alpha, one in a hundred, is smaller than the p-value, almost two in a hundred, so the sample is not rare enough to meet this test's standard.

At the 5 percent level the same p-value would reject, which is why the conclusion has to name its level. Without it, a reader cannot tell whether 0.018 passed or failed.

One wrong option reaches the right decision and then says too much. That pattern, right decision with wrong meaning, is the one to watch for in your own writing.

64. Check: a full test, start to finish

Check

This one needs the whole procedure. Sketch the curve before choosing.

Find the standard error first, then the z-score, then decide which tail the alternative points to.

Check your understanding

A bottler claims its bottles hold more than 500 mL on average. Fills are normal with sigma 4 mL. A sample of 16 bottles averages 502 mL. At alpha 0.05, what are the p-value and conclusion?

  • A. z = 2, p-value ≈ 0.0228: reject; sufficient evidence that the mean exceeds 500 mL (correct)
  • B. z = 0.5, p-value ≈ 0.3085: do not reject; not sufficient evidence
  • C. z = 2, p-value ≈ 0.0455: reject; sufficient evidence that the mean differs from 500 mL
  • D. z = 2, p-value ≈ 0.0228: reject; the mean is 502 mL

Answer: A

Why: The standard error is 4 over the square root of 16, which is 1. So z is 2 over 1, which is 2, and the right tail beyond 2 is 0.0228. Since 0.05 is greater than 0.0228, reject: there is sufficient evidence at the 5 percent level that the mean fill exceeds 500 mL.

Why B tempts people
That uses sigma instead of the standard error, treating the mean of 16 bottles as if it were a single bottle.
Why C tempts people
That doubles the tail as if the test were two-sided. The claim says more than, so only the right tail counts.
Why D tempts people
The decision is right but the conclusion states the sample mean as the population mean.

In context: if the bottles really averaged 500 mL, a sample of 16 averaging 502 mL or more would turn up only about 23 times in 1,000, so the claim of more than 500 mL is supported at the 5 percent level.

\[ \frac{4}{\sqrt{16}} = 1, \quad z = \frac{502 - 500}{1} = 2, \quad P(Z \ge 2) = 0.0228 \]

Each wrong option is a different step of the pattern done wrongly: the standard error, the tail, and the conclusion's wording. If you missed this one, find which step and redo just that step.

65. Explain the rare-event argument in your own words

Explain it to yourself

The whole section rests on one piece of reasoning. Explaining it without symbols is the best test of whether it has landed.

Discussion prompt

Explain, in three or four sentences and without any formulas, why a small p-value is a reason to reject the null hypothesis, but a large p-value is not a reason to accept it.

A strong answer names the condition: the probability was computed as if the null were true. A very strong one names the asymmetry: unlikely data rule an assumption out, but likely data rule nothing in.

If your answer used the words proves or accepts anywhere, rewrite that sentence. The strongest words a test earns are sufficient evidence and doubt.

66. Exit ticket: the commute that got longer

Exit ticket

A city reports that the mean commute is 25 minutes, with a standard deviation of 6 minutes. A transport researcher thinks commutes have become longer. A random sample of 36 commuters averages 27.1 minutes. The level is 0.05.

Discussion prompt

State the hypotheses, find the p-value, make the decision, and write the conclusion in context.

Check yourself against the pattern. The alternative came from the word longer. The standard error was 1 because 6 divided by the square root of 36 is 1. The tail was the right one. The decision compared alpha with the p-value in the right order, and the conclusion named the level and the population mean.

If your p-value was 0.0357, you shaded both tails; if it was near 0.36, you used the standard deviation of single commutes instead of the standard error of the mean.

67. Recap

Recap

A hypothesis test runs on the rare-event argument: assume the null, and if the sample would be very unlikely under that assumption, doubt the assumption. The p-value measures how unlikely, and a preset alpha turns the measurement into a decision.

ideastatementread it as
rare-event argumentunlikely under H₀ ⇒ doubt H₀the data are real; H₀ is only an assumption
p-valueP(as extreme or more | H₀ true)a tail area on the null's curve
directionHₐ: < left, > right, ≠ both tailsthe alternative chooses the tail
decision ruleα > p-value ⇒ reject H₀α ≤ p-value ⇒ do not reject
alphaset before the data; 0.05 if none givenP(Type I error) = α
notP(H₀ | data), effect size, proofconditional on H₀; significant ≠ important
conclusionat level α, (not) sufficient evidence that Hₐin context, about the population

The arithmetic was Chapter 7's: a standard error, a z-score, a tail area. The new skills are the logic around it: keeping the if-the-null-is-true condition attached to every p-value, comparing with alpha in the right direction, and writing a conclusion that claims no more than the evidence allows.

Section 9.5 practises full tests of all three shapes, left, right and two-tailed. The Introductory Business Statistics text works the same procedure in its §9.4, with business examples, if you want more practice in a different setting.

OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion §9.4, pp. 467-469 — the whole section

OpenStax Introductory Business Statistics 2e, §9.4 Full Hypothesis Test Examples §9.4 Full Hypothesis Test Examples — the same procedure in business contexts

Sources

  1. OpenStax Introductory Statistics 2e, §9.4 Rare Events, the Sample, Decision and Conclusion — Illowsky & Dean, OpenStax / Rice University, CC BY 4.0, pp. 466-470
  2. OpenStax Introductory Business Statistics 2e, §9.4 Full Hypothesis Test Examples — Illowsky & Dean, OpenStax / Rice University, CC BY 4.0

Want this taught 1-on-1? Alexander tutors Statistics — $55/session, free consultation.

Book on Wyzant · Text (657) 465-8108