9.2 Outcomes and the Type I and Type II Errors

A hypothesis test decides on incomplete evidence, so it can be wrong in two distinct ways. Crossing the two possible decisions with the two possible truths gives four outcomes: two of them correct, and two of them errors. A Type I error is rejecting the null hypothesis when it is true, and its probability is called alpha. A Type II error is failing to reject the null when it is false, and its probability is called beta. Both should be as small as possible and neither is ever zero, and they trade against each other — moving the decision boundary to reduce one increases the other. Only a larger sample reduces both at once. The probability of correctly rejecting a false null is one minus beta, and is called the power of the test. Which of the two errors matters more is a judgement about consequences that the statistics cannot supply.

Subject: Statistics · 66 slides · symbolic lesson

Open the interactive version of this deck

What this lesson covers

The lesson, slide by slide

1. Outcomes and the Type I and Type II Errors

Title

Statistics · §9.2

A test decides without seeing the truth, so it can be wrong in two different ways

2. By the end of this lesson you can

Objectives

Section 9.1 set up the two hypotheses. Before any test is run, this section asks a sobering question: when the test is finished and you have made your decision, how could that decision be wrong, and how often?

  1. List the four possible outcomes of a hypothesis test and name the two that are errors.
  2. Define a Type I error and its probability α, in the words of a stated null hypothesis.
  3. Define a Type II error and its probability β, in the same terms.
  4. Judge which error carries the greater consequence in a given situation, by the action it leads to.
  5. Define the power of a test, 1 − β, and say what increases it.

The book's own examples here are all words: rock-climbing gear, an accident victim, a fertility clinic, a cancer drug. You will also see the errors as areas under curves you already know from Chapter 7, because that picture is what makes the trade-off between them obvious.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, pp. 464-466 — the section these objectives come from

3. What 9.1 gave you

Warm-up

Every error in this section is described in terms of a null hypothesis, so start by writing one. Try It 9.3 asks whether it takes fewer than 45 minutes, on average, to teach a lesson plan.

Discussion prompt

State the null and alternative hypotheses for the lesson-plan question. Then say, in words, what it would mean to reject the null and what it would mean not to reject it.

The lesson-plan example is a good one to keep, because both hypotheses are about the same population mean, the average teaching time over all lessons, not about any single lesson. Every error in this section is likewise a statement about a population, judged from a sample.

Keep two facts from 9.1 in view. A test ends in exactly one of two decisions, reject the null or do not reject it. And the decision is made from a sample, so it is never a certainty about the population. The rest of this lesson is about the gap between the decision and the truth.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.1, p. 463 — Try It 9.3

4. The four outcomes

Section

Part 1

5. Two decisions crossed with two truths

Concept

Two things are in play when a test finishes. One is the truth about the population: the null hypothesis is either actually true or actually false. You never get to see this. The other is your decision, which you do see: reject the null, or do not reject it.

Cross the two and you get four possible outcomes. The book sets them out as Table 9.2, drawn here as a grid. The truth chooses the column; your test chooses the row.

Figure (svg): A two-by-two grid. Columns: H0 is actually true, H0 is actually false. Rows: do not reject H0, reject H0. The top-left cell is green, correct outcome, probability 1 minus alpha. The top-right cell is red, Type II error, probability beta. The bottom-left cell is red, Type I error, probability alpha. The bottom-right cell is green, correct, power, probability 1 minus beta. Under each column: this column's probabilities add to 1.

The truth picks the column and the test picks the row. The two errors sit on the off-diagonal, and they are in different columns, so they can never happen together.

Two cells are good news: you keep a null that is true, or you reject a null that is false. The other two are errors. Rejecting a true null is called a Type I error; failing to reject a false null is called a Type II error.

\[ \text{Type I: reject } H_0 \text{ when } H_0 \text{ is true} \]

\[ \text{Type II: do not reject } H_0 \text{ when } H_0 \text{ is false} \]

Notice that the two errors sit in different columns. In any one real situation the null is either true or false, so only one of the two errors is even possible, and you cannot know which one you are exposed to.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — Table 9.2 and the four outcomes

6. Decoding alpha, beta and power

Notation

Each error has a probability, and the book names them with Greek letters. Read each symbol as a probability about the test procedure, worked out before you see any data.

Annotate

On: \( \alpha = P(\text{Type I}) \qquad \beta = P(\text{Type II}) \qquad 1 - \beta \)

  • Alpha: the probability of rejecting the null when the null is true. You choose it before collecting data; it is also called the significance level.
  • A false alarm: the test announces an effect that is not there.
  • Beta: the probability of not rejecting the null when the null is false. It depends on how false the null is.
  • A miss: a real effect is there and the test does not detect it.
  • The power of the test: the probability of correctly rejecting a false null. You want it as close to 1 as possible.

The book adds that alpha and beta should be as small as possible, because they are probabilities of mistakes, and that they are rarely zero. A test that could never make a Type I error would have to never reject anything, and then it would miss every real effect.

\[ \alpha = P(\text{reject } H_0 \mid H_0 \text{ true}) \]

\[ \beta = P(\text{do not reject } H_0 \mid H_0 \text{ false}) \]

The vertical bar is read as given. Each probability is conditional on one column of the grid, which is the idea the next slide builds on.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — α, β and the Power of the Test

7. Each probability lives in one column

Concept

Look at the grid one column at a time. If the null is true, only the left column can happen, and the test either keeps it or rejects it. Those two outcomes cover everything, so their probabilities add to one.

\[ P(\text{keep } H_0 \mid H_0 \text{ true}) + P(\text{reject } H_0 \mid H_0 \text{ true}) = (1 - \alpha) + \alpha = 1 \]

The right column works the same way. If the null is false, the test either misses it, with probability beta, or catches it, with probability called the power.

\[ \beta + (1 - \beta) = 1 \]

Here is the consequence that catches people out: alpha and beta do not add to one. They are probabilities in different columns, computed under different assumptions about the truth. A test with alpha 0.05 does not have beta 0.95; its beta could be 0.36, or 0.01, or 0.80, depending on the sample size and how far the truth is from the null.

\[ \alpha + \beta \ne 1 \quad \text{in general} \]

A quick way to remember it: alpha and power are both probabilities of rejecting, but under different truths. Alpha is the rejection rate when nothing is there; power is the rejection rate when something is. A good test has the first small and the second large.

So whenever you write a probability for an outcome, say which column it is conditioned on. That habit prevents most of the misreadings in this chapter.

8. A courtroom runs the same test

Intuition

A criminal trial is a hypothesis test that everyone already understands. The defendant is presumed innocent, so innocence is the null hypothesis. The prosecution must bring evidence strong enough to reject it.

defendant is innocent (H₀ true)defendant is guilty (H₀ false)
verdict: not guilty (do not reject)correctType II: a guilty person goes free
verdict: guilty (reject H₀)Type I: an innocent person is convictedcorrect

The legal system deliberately makes the Type I error rare. Beyond a reasonable doubt is a very small alpha: society has decided that convicting an innocent person is worse than letting a guilty one go. The price of that choice is more Type II errors.

The courtroom also explains the careful wording of a test's conclusion. A jury returns not guilty, never innocent. Not guilty means the evidence was not strong enough, exactly as do not reject the null means the data were not strong enough. It is not a finding that the null is true.

The jury's standard of proof is the courtroom's alpha. Lowering it, say to the balance of probabilities used in civil cases, convicts more of the guilty and more of the innocent at once. The trade-off you will meet in Part 5 is exactly that.

Keep this example in your pocket. Whenever the abstract labels blur, ask which is the innocent person convicted and which is the guilty person freed.

9. Correct decision, or which error?

Sorting

Each line below describes a decision and the truth behind it. Place each one in its cell of the grid. Read the truth first, which picks the column, then the decision, which picks the row.

Sort into buckets

Sort each outcome into its cell.

correct outcome
A drug that works is approved after a trial (H₀: the drug has no effect); A screening test clears a patient who is healthy (H₀: the patient is not sick)
Type I error: a true null rejected
A smoke alarm sounds while toast is browning normally (H₀: there is no fire); A quality check stops a batch that met specification (H₀: the batch meets spec)
Type II error: a false null kept
A jury acquits a defendant who did commit the crime (H₀: innocent); A spam filter lets a phishing email into the inbox (H₀: the email is legitimate)
ok
The approved drug really works, so rejecting no effect was right. The healthy patient was cleared, so keeping not sick was right.
t1
There was no fire and the batch was fine, so each null was true, and the alarm and the stop both rejected it. Those are false alarms.
t2
The defendant was guilty and the email was phishing, so each null was false, yet each was kept. Those are misses.

The quick test for a Type I error is that something was declared, an alarm, a stop, a conviction, when nothing was really there. A Type II error is the reverse: something was really there and nothing was declared.

If you put the spam-filter line with the false alarms, re-read the null. The null is that the email is legitimate, and the filter kept it, so the null was not rejected. Since the email was in fact phishing, the null was false: a miss.

10. Try It 9.8: four statements, four cells

Matching

Try It 9.8 assumes the null hypothesis that the percentage of adults with jobs is at least 88 percent, and gives four statements. Match each to the outcome it describes. Find the decision and the truth in each sentence before you match.

Match the pairs

  • s1. Not to reject H₀ (at least 88%) when the percentage is actually less than 88%
  • s2. Not to reject H₀ (at least 88%) when the percentage is actually at least 88%
  • s3. Reject H₀ (at least 88%) when the percentage is actually at least 88%
  • s4. Reject H₀ (at least 88%) when the percentage is actually less than 88%
  • t2. Type II error
  • c1. correct: a true null kept
  • t1. Type I error
  • c2. correct: a false null rejected (power)

Why: The null says at least 88 percent. When the truth is at least 88 percent the null is true, so rejecting it is a Type I error and keeping it is correct. When the truth is less than 88 percent the null is false, so keeping it is a Type II error and rejecting it is the correct outcome the power measures.

\[ H_0: p \ge 0.88 \qquad H_a: p < 0.88 \]

Every statement in the book has the same two halves, a decision and then a when-the-truth-is clause. That structure is not decoration: an error cannot be named without both halves, because the same decision is right in one column and wrong in the other.

So the two errors in this Try It are the statement that rejects while the percentage really is at least 88, a Type I error, and the statement that does not reject while it is really below 88, a Type II error.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 466 — Try It 9.8

11. The Type I error

Section

Part 2

12. A Type I error is a false alarm

Concept

A Type I error rejects a null hypothesis that is actually true. In plain words, the test claims to have found something, an effect, a difference, a danger, that is not there. That is why it is often called a false alarm or a false positive.

Type I error — Rejecting the null hypothesis when it is in fact true. Its probability is alpha, the significance level of the test.

\[ \alpha = P(\text{reject } H_0 \mid H_0 \text{ true}) \]

What makes alpha special is that you set it. Before collecting any data, you decide how much false-alarm risk you will tolerate, commonly 0.05 or 0.01, and the test's cutoff is built to match. Sections 9.3 to 9.5 show how; for now, picture a cutoff on the sampling distribution.

Why not set alpha to zero? Because the only test that never rejects a true null is one that never rejects anything. Such a test would also never detect a real effect, so its power would be zero. Some false-alarm risk is the price of being able to detect anything at all.

A Type I error is also the error a researcher is tempted towards. A significant result is exciting and publishable, so the rule that alpha is fixed in advance, and small, is the discipline that keeps excitement from manufacturing findings.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — α = probability of a Type I error

13. Setting the cutoff from alpha: a clam-toxin test

Worked example

Try It 9.7 describes testing clams for red-tide toxin, with harvesting banned when the mean level exceeds 800 micrograms per kilogram. Make it numerical with assumptions of our own: readings vary with standard deviation 100, and the inspectors average 16 samples.

\[ H_0: \mu \le 800 \qquad H_a: \mu > 800 \qquad n = 16, \ \sigma = 100 \]

The test will reject the null, and ban harvesting, when the sample mean is far enough above 800. To choose how far, work at the boundary of the null, where the true mean is exactly 800: that is where a false alarm is most likely.

Standard error of the mean

Why: the spread of the sample mean of 16 readings, from Chapter 7.

\[ \sigma_{\bar{x}} = \frac{100}{\sqrt{16}} = \frac{100}{4} = 25 \]

Find the z with 0.05 above it

Why: alpha is the right-tail area under the null's curve.

\[ z_{0.95} = 1.6449 \]

Convert to micrograms

Why: start at 800 and walk 1.6449 standard errors up.

\[ c = 800 + 1.6449(25) \]

Multiply and add

Why: the cutoff for the sample mean.

\[ c = 800 + 41.12 = 841.1 \]

Figure (svg): A bell curve for the mean toxin level of 16 samples, centred at 800 with standard error 25, on an axis from 710 to 910. A dashed yellow line at 841.1 is the cutoff. The thin right tail beyond it is shaded red and labelled alpha equals 0.05. Above the frame, the region left of the cutoff is labelled do not reject, the region right of it reject.

Alpha is an area under the curve drawn as if the null were true. Five percent of that curve lies beyond 841.1, so a perfectly safe area still triggers a ban five times in a hundred.

So the rule is: ban harvesting if the mean of 16 samples is above 841.1. In context, if an area's clams truly average 800, a sample this high happens only 5 percent of the time, and every such ban is a Type I error.

Check the alpha this cutoff delivers

Why: standardise 841.1 under the null and read the right tail.

\[ P\left(Z > \frac{841.1 - 800}{25}\right) = P(Z > 1.645) = 0.0500 \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 465 — Try It 9.7 (the numbers are the deck's own)

14. Decoding the cutoff formula

Notation

The clam-test cutoff came from one line, and the same line will build every cutoff in this lesson. Each piece of it carries a decision you have already made, so read it piece by piece.

Annotate

On: \( c = \mu_0 + z_{1-\alpha}\,\frac{\sigma}{\sqrt{n}} \)

  • The cutoff for the sample mean: reject the null when the sample mean lands beyond it.
  • The null value, 800 here. The cutoff is built as if the null were true, at its boundary.
  • How many standard errors out the cutoff sits: 1.6449 when alpha is 0.05, 2.3263 when alpha is 0.01.
  • The standard error of the sample mean, from Chapter 7: 100 over 4, which is 25, for 16 samples.
  • Plus because the alternative points up, more than 800. For an alternative pointing down the cutoff is below the null value.

Notice what is missing from the formula: the true mean. The cutoff is fixed entirely by the null value, the chosen alpha and the sample size. That is why alpha is under your control and beta is not: the truth only enters later, when you ask how often the real sample mean crosses the line.

\[ c = 800 + 1.6449(25) = 841.1 \]

Two of the four ingredients are levers. Lowering alpha makes the z bigger and pushes the cutoff out; raising n shrinks the standard error and pulls it in. Part 5 pulls both levers and watches the errors respond.

15. How many false alarms in 100 safe areas?

Estimation

Suppose 100 areas are tested with the rule from the last slide, and every one of them is truly safe, with a mean toxin level of exactly 800. Before looking at the simulation, estimate how many bans the rule will issue.

Predict first

About how many of the 100 truly safe areas will the rule ban?

  • about 5
  • none, because every area is safe
  • about 50
  • about 95

Correct: about 5

Why: Each safe area has a 0.05 chance of producing a sample mean above the cutoff, because that is how the cutoff was built. Over 100 independent tests that averages 5 false alarms, though any one batch of 100 can give a few more or a few fewer.

Figure (svg): One hundred dots, one per simulated test, plotted by test number from 1 to 100 against the sample mean of 16 toxin readings, all drawn from areas whose true mean is exactly 800. Most dots scatter between about 750 and 850. A dashed yellow line at 841.1 marks the cutoff; the 6 dots above it are red, each a Type I error.

Every one of these areas is safe, so every red dot is a false alarm. This run produced 6 of them in 100 tests; the long-run rate is alpha, 5 in 100, and no single run hits it exactly.

The simulation drew 100 genuine samples of 16 readings from a population with mean 800, and 6 of them landed above 841.1. Each of those red dots would have banned harvesting from a safe area, and nothing in the data marks them as mistakes.

The option none, because every area is safe, is the tempting one. It confuses the truth with the evidence. The test never sees the truth; it sees a sample, and a sample from a safe area is sometimes high just by chance. Alpha is the long-run rate at which that happens.

The count also varies from batch to batch. The number of false alarms in 100 tests of true nulls is itself binomial, with 100 trials and probability 0.05, so counts anywhere from about 1 to 9 are ordinary.

\[ \text{expected false alarms} = 100 \times 0.05 = 5 \]

16. Alpha is chosen before the data arrive

Concept

Alpha is also called the significance level of the test, and it is the only error probability you control directly. You write it down before you collect data, which matters: choosing it afterwards, to make your result come out significant, defeats its purpose.

Smaller alpha means a cutoff further from the null value. For the clam test, here is where the cutoff lands for three common choices of alpha, each computed exactly as on the earlier slide.

significance level αz with that area above itcutoff c = 800 + z(25)
0.101.2816832.0
0.051.6449841.1
0.012.3263858.2

Read down the table. Demanding more evidence before a ban pushes the cutoff from 832 up to 858. That protects safe areas from false bans, but it also means a genuinely toxic area must show a higher sample mean before it is caught.

\[ \alpha \downarrow \quad \Rightarrow \quad c \uparrow \]

That second effect is the Type II error, which Part 3 measures. Hold onto the arrow above: every move that makes false alarms rarer, with the sample size fixed, makes misses more common.

17. Example 9.5: Navah's rock-climbing equipment

Worked example

Suppose the null hypothesis is that Navah's rock-climbing equipment is safe. Describe both errors in context, say what alpha and beta mean here, and decide which error has the greater consequence.

\[ H_0: \text{the equipment is safe} \]

Name the Type I error

Why: reject a true null: decide the gear may not be safe when it really is.

\[ \text{Type I: thinks not safe} \mid \text{in fact safe} \]

Name the Type II error

Why: keep a false null: decide the gear may be safe when it is not.

\[ \text{Type II: thinks safe} \mid \text{in fact not safe} \]

Each description has the decision first and the truth second, the truth always introduced by when, in fact. Without the second half, the sentence would just describe a decision, and the same decision is correct in the other column.

Read alpha and beta in context

Why: each is the probability of one of those two sentences.

\[ \alpha = P(\text{thinks not safe} \mid \text{safe}), \quad \beta = P(\text{thinks safe} \mid \text{not safe}) \]

Follow each error to its action

Why: what Navah does next decides how much it costs.

\[ \text{Type I} \to \text{buys new gear}; \quad \text{Type II} \to \text{climbs on unsafe gear} \]

Figure (svg): A two-row flow diagram for Navah's climbing gear. Top row: the test says may not be safe, so she rejects H0; she replaces the gear or stays home; if she is wrong, a yellow box reads Type I: money spent, a day lost. Bottom row: the test says may be safe, so she keeps H0; she climbs on it; if she is wrong, a red box reads Type II: a fall, serious injury.

Follow each decision to what Navah then does. The Type I path ends in an inconvenience; the Type II path ends on a rock face.

The book's verdict: the error with the greater consequence is the Type II error, because if Navah thinks her equipment is safe she will go ahead and use it. A Type I error costs money and a day; a Type II error can cost her life.

Alpha and beta here are not numbers you could compute: nobody has a sampling distribution for Navah's inspection. They are still meaningful as rates. If Navah inspected a thousand safe harnesses, alpha is the fraction she would wrongly condemn; if she inspected a thousand faulty ones, beta is the fraction she would wrongly trust.

Check each error against the grid

Why: Type I must sit in the H0-true column and Type II in the H0-false column.

\[ \text{Type I} \mid \text{safe (true)}, \quad \text{Type II} \mid \text{not safe (false)} \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — Example 9.5

18. Before you describe an error, write the null in words

Step zero

Every description of an error is built from the null hypothesis, so the first move is always the same: write the null as a plain sentence about the situation. Try It 9.5 gives the null that the blood cultures contain no traces of pathogen X.

Discussion prompt

Before naming any error, write the two halves you will need: what does rejecting this null mean in words, and what does it mean for the null to be true?

Once you have those phrases, the template does the rest. A Type I error is the reject phrase, joined by when in fact, to the null-true phrase. A Type II error is the do-not-reject phrase joined to the null-false phrase.

\[ \text{Type I} = \text{[reject phrase] when, in fact, [}H_0\text{ true phrase]} \]

Skipping this step is how a description ends up backwards. If you start writing before you know what rejecting means in context, a no-pathogen null is easily flipped in your head into a pathogen null.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — Try It 9.5

19. Try It 9.5: the blood cultures

Worked example

The null hypothesis is that the blood cultures contain no traces of pathogen X. State the Type I and Type II errors in context, then say which matters more.

\[ H_0: \text{no pathogen X in the cultures} \]

Type I: reject a true null

Why: the lab reports the pathogen, but the cultures are clean.

\[ \text{thinks pathogen present} \mid \text{in fact absent} \]

Type II: keep a false null

Why: the lab reports clean cultures, but the pathogen is there.

\[ \text{thinks no pathogen} \mid \text{in fact present} \]

Now follow each to an action. After a Type I error the patient is treated for an infection they do not have, which means unnecessary drugs, perhaps side effects, and a retest that will likely clear it up.

After a Type II error the patient goes untreated for an infection they do have, and nothing prompts a second look. For a dangerous pathogen that is the error with the greater consequence.

Name the alpha and beta

Why: the probabilities of those two sentences.

\[ \alpha = P(\text{reports pathogen} \mid \text{clean}), \quad \beta = P(\text{reports clean} \mid \text{pathogen}) \]

Check against the template

Why: each error is a decision plus a truth from the opposite column.

\[ \text{Type I: reject} + H_0 \text{ true}, \quad \text{Type II: keep} + H_0 \text{ false} \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, pp. 464-465 — Try It 9.5

20. Trap: an error with no truth in it

Trap

The trap

Asked for the Type I error in Example 9.5, a first attempt writes only the decision:

\[ \text{Type I: Navah thinks her} \]

\[ \text{equipment may not be safe.} \]

Incomplete. That sentence describes a decision, and the decision is correct if the gear really is unsafe.

The fix

An error is a mismatch between the decision and the truth, so both must be named. Add the when, in fact clause from the correct column.

\[ \text{thinks not safe} \]

\[ \text{when, in fact, it is safe} \]

The book writes every one of its errors this way, and a marker will look for the second half. A useful self-check: if your sentence could describe a correct decision, it is missing its truth clause.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — Example 9.5

21. Errors for the driver's-test null

Fill the middle

Try It 9.4 tests whether more than 40 percent of drivers pass the state test on the first try, so the null is that at most 40 percent do. Fill in the missing words in the two error descriptions. Each blank is one word.

Fill in the blanks

Type I error: concluding that more than 40% pass on the first try when, in fact, at most 40% pass. Type II error: not concluding that more than 40% pass when, in fact, more than 40% pass. The probability of the first error is called alpha.

Why: Rejecting the null means concluding the alternative, that more than 40 percent pass, so the Type I error makes that conclusion while the null is true. The Type II error fails to conclude it while the alternative is true, that is, while more than 40 percent really pass. The probability of a Type I error is alpha.

\[ H_0: p \le 0.40 \qquad H_a: p > 0.40 \]

Both blanks in the descriptions are the same word, and that is the pattern worth noticing. The rejection always concludes the alternative, and the Type II error is always the case where the alternative is true but was not concluded.

Once you see that, describing errors for a new null is mechanical: rejection means the alternative, and the truth clause comes from whichever hypothesis is actually correct.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.1, p. 463 — Try It 9.4

22. The Type II error

Section

Part 3

23. A Type II error is a missed detection

Concept

A Type II error fails to reject a null hypothesis that is actually false. Something real is going on, a toxic area, an effective drug, a sick patient, and the test does not notice. It is a miss, or a false negative.

Type II error — Failing to reject the null hypothesis when it is in fact false. Its probability is beta.

\[ \beta = P(\text{do not reject } H_0 \mid H_0 \text{ false}) \]

There is a catch hidden in that definition. To compute alpha you assume the null is true, and the null names one value, 800 in the clam test. To compute beta you assume the null is false, but false could mean a true mean of 801, or 850, or 1000.

So beta is not one number. It depends on how false the null is: a truth far from the null is easy to detect and gives a small beta, while a truth just past the null is very hard to detect and gives a large one. Whenever you see a value of beta, it belongs to one particular alternative value.

Where a Type I error is the researcher's temptation, a Type II error is the researcher's disappointment: a real effect, a study that did not find it, and a report that says nothing significant. It is quieter than a false alarm, and far more common than most people realise.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — β = probability of a Type II error

24. Two curves, one cutoff

Picture it

Put the clam test's two worlds on one axis. The blue curve is how the sample mean behaves if the null is true and the clams average 800. The orange curve is how it behaves if they actually average 850, well over the limit.

Figure (svg): Two bell curves of the same width on one axis of mean toxin level: a blue one centred at 800, labelled H0 true, and an orange one centred at 850, labelled truth mu equals 850. A dashed yellow cutoff stands at 841.1. The blue curve's tail right of the cutoff is shaded red, alpha 0.05. The orange curve's area left of the cutoff is shaded blue, beta 0.36.

One cutoff, two curves, two errors. Alpha is the part of the null's curve on the reject side; beta is the part of the true curve on the do-not-reject side. They are areas under different curves.

The cutoff at 841.1 splits the axis into a do-not-reject side and a reject side. Alpha is the blue curve's area on the reject side, red in the picture. Beta is the orange curve's area on the do-not-reject side, shaded blue: samples from a toxic area that still come out below 841.1.

Look at how much of the orange curve sits left of the cutoff. More than a third of it does. So a test with only 16 samples misses a genuinely toxic area at 850 about 36 times in 100, while its false-alarm rate is only 5 in 100.

Also notice what the picture does not show: which curve is real. In practice you get one sample mean, a single point somewhere on this axis, and you never learn which curve produced it. The two shaded areas describe the test's behaviour in each world, not the world you are in.

This is the single most useful picture in the chapter. Every claim in the rest of this lesson, about moving the cutoff, changing the sample size or changing the truth, is a claim about how these two shaded areas change.

25. Beta and power when the clams average 850

Worked example

Use the clam test: 16 samples, standard deviation 100, ban if the sample mean exceeds 841.1. Suppose an area's clams really average 850 micrograms per kilogram. Find the probability the test misses it, and the power.

\[ \bar{X} \sim N(850,\ 25) \text{ if } \mu = 850, \quad \text{miss if } \bar{x} \le 841.1 \]

The cutoff does not change when the truth does: it was set from the null, before any data. What changes is the curve the sample mean is actually drawn from, now centred at 850.

Standardise the cutoff under the truth

Why: distance from 850, in standard errors of 25.

\[ z = \frac{841.12 - 850}{25} \]

Divide

Why: the cutoff sits a third of a standard error below the true mean.

\[ z = \frac{-8.88}{25} = -0.355 \]

Area to the left is beta

Why: samples that fall on the do-not-reject side.

\[ \beta = P(Z < -0.355) = 0.3612 \]

Power is the rest of that curve

Why: one minus beta.

\[ 1 - \beta = 1 - 0.3612 = 0.6388 \]

In context: if an area's clams truly average 850 micrograms per kilogram, a 16-sample test at the 0.05 level leaves harvesting open about 36 percent of the time, and catches the toxic area only about 64 percent of the time.

Check by computing the power directly

Why: power is the true curve's area right of the cutoff, so it must match one minus beta.

\[ P(\bar{X} > 841.12 \mid \mu = 850) = P(Z > -0.355) = 0.6388 \quad \checkmark \]

26. Decoding the beta formula

Notation

The worked slide computed beta in three moves. Packed into one line, they read like this; each symbol points back to a choice or an assumption, and the most important one is the true mean.

Annotate

On: \( \beta(\mu) = P\left(Z < \frac{c - \mu}{\sigma/\sqrt{n}}\right) \)

  • Beta written as a function of the true mean: a different miss probability for every possible truth.
  • The same cutoff as before, fixed by the null value, alpha and n. It does not move when the truth does.
  • The true mean you are asking about, such as 850. It must lie in the alternative, above 800, or there is nothing to miss.
  • How far the cutoff sits from the truth. Negative here, because the cutoff is below 850.
  • A left tail: samples from the true curve that fall on the do-not-reject side of the cutoff.

Compare it with alpha. Alpha is also a tail area beyond the same cutoff, but under the null's curve, so its formula has the null value where this one has the true mean. Same cutoff, different curve, different tail.

\[ \alpha = P\left(Z > \frac{c - 800}{25}\right), \quad \beta(850) = P\left(Z < \frac{c - 850}{25}\right) \]

For an alternative pointing down, as in the exit ticket at the end, both inequalities flip: the reject side is below the cutoff, so beta becomes a right tail of the true curve.

27. Which toxic area is hardest to catch?

Prediction

Keep the same test, cutoff 841.1. Three areas are all over the 800 limit, so for each of them the null is false. Predict which one the test is most likely to miss.

Predict first

For which true mean is beta, the chance of missing it, largest?

  • μ = 820, just over the limit
  • μ = 850
  • μ = 900, far over the limit
  • the same for all three, since the null is false in each

Correct: μ = 820, just over the limit

Why: The closer the truth is to 800, the more its curve overlaps the null's curve, so the more of it falls below the cutoff. Beta is about 0.80 at 820, 0.36 at 850 and only 0.009 at 900.

\[ \beta(820) = \Phi\left(\tfrac{841.12 - 820}{25}\right) = \Phi(0.845) = 0.80 \]

\[ \beta(900) = \Phi\left(\tfrac{841.12 - 900}{25}\right) = \Phi(-2.355) = 0.009 \]

The option that says all three are the same treats beta as a property of the test alone. It is not: beta changes with the truth. A small violation of the null is a small signal, and small signals are easily lost in sampling noise.

That is reassuring in one way and sobering in another. Badly toxic areas are almost always caught; areas just over the limit are usually missed.

28. Beta is a function of the truth

Concept

Since beta depends on the true mean, the honest way to report it is as a table, or a curve, over several possible truths. Here is the clam test with 16 samples at the 0.05 level, each row computed exactly as on the worked slide.

true mean μz = (841.12 − μ)/25β = P(miss)power = 1 − β
8200.8450.8010.199
8250.6450.7410.259
850−0.3550.3610.639
875−1.3550.0880.912
900−2.3550.0090.991

Now push the truth down toward the limit itself. As the true mean approaches 800 from above, the true curve slides onto the null's curve, and the chance of missing approaches the chance of not rejecting under the null.

\[ \mu \to 800^{+} \quad \Rightarrow \quad \beta \to 1 - \alpha = 0.95 \]

So no test can reliably detect an effect that is barely there. This is why a study must decide in advance what size of effect it cares about, here perhaps a mean of 850, before it can say anything about beta at all.

29. Example 9.6: the accident victim

Worked example

Suppose the null hypothesis is that the victim of an automobile accident is alive when they arrive at the emergency room. Describe both errors, give alpha and beta in words, and say which error has the greater consequence.

\[ H_0: \text{the victim is alive} \]

Type I: reject a true null

Why: the crew decides the victim is dead, but they are alive.

\[ \text{thinks dead} \mid \text{in fact alive} \]

Type II: keep a false null

Why: the crew treats the victim as alive, but they have died.

\[ \text{thinks alive} \mid \text{in fact dead} \]

A note on the book's wording. It states the Type II error as the crew does not know if the victim is alive when, in fact, the victim is dead. The clean form is that the crew continues to treat the victim as alive, because not rejecting the null means acting as if it were true.

Alpha and beta in words

Why: each is the probability of one of those errors.

\[ \alpha = P(\text{thinks dead} \mid \text{alive}), \quad \beta = P(\text{thinks alive} \mid \text{dead}) \]

Follow each error to its action

Why: the action decides the cost.

\[ \text{Type I} \to \text{no treatment}; \quad \text{Type II} \to \text{treatment continues} \]

After a Type I error the crew stops treating someone who could be saved. After a Type II error they work on a patient who cannot be helped, which costs time and effort but harms no one. The Type I error has the greater consequence.

Check against Example 9.5

Why: here the worse error is Type I, there it was Type II; the label alone never decides.

\[ \text{Ex 9.5: Type II worse}, \quad \text{Ex 9.6: Type I worse} \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 465 — Example 9.6

30. Try It 9.6: a patient who is not sick

Worked example

Suppose the null hypothesis is that a patient is not sick. Which type of error has the greater consequence? Describe both before judging.

\[ H_0: \text{the patient is not sick} \]

Type I: reject a true null

Why: the doctor decides the patient is sick, but they are healthy.

\[ \text{thinks sick} \mid \text{in fact healthy} \]

Type II: keep a false null

Why: the doctor decides the patient is fine, but they are sick.

\[ \text{thinks not sick} \mid \text{in fact sick} \]

Follow the actions. A Type I error leads to treatment, or more tests, for a healthy person: worry, expense, and side effects, but usually a later test reveals the mistake.

A Type II error sends a sick patient home. The illness goes untreated, may get worse, and nothing prompts anyone to look again. For most illnesses that is the error with the greater consequence, so the answer is the Type II error.

This is the same shape as the blood-culture case, and for the same reason: both nulls say nothing is wrong, so the Type II error is the one that misses a real problem.

Check against the structure of the null

Why: a null of the form nothing is wrong puts the missed problem in the Type II cell.

\[ H_0 = \text{nothing wrong} \Rightarrow \text{Type II} = \text{missed problem} \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 465 — Try It 9.6

31. Example 9.8: a cure rate of at least 75 percent

Worked example

An experimental drug claims a cure rate of at least 75 percent for males with prostate cancer. Describe both errors in context and decide which is more serious.

\[ H_0: p \ge 0.75 \qquad H_a: p < 0.75 \]

The claim contains the equality, at least, so it is the null, and the alternative is that the cure rate falls short. Rejecting the null therefore means concluding the drug is worse than claimed.

Type I: reject a true null

Why: decide the cure rate is below 75 percent when it really is at least 75.

\[ \text{believes } p < 0.75 \mid \text{in fact } p \ge 0.75 \]

Type II: keep a false null

Why: accept the claim when the cure rate is really below 75 percent.

\[ \text{believes } p \ge 0.75 \mid \text{in fact } p < 0.75 \]

Follow each error to the patient's choice

Why: the belief drives whether the drug is used.

\[ \text{Type I} \to \text{passes over a good drug}; \quad \text{Type II} \to \text{relies on a weaker drug} \]

The book judges the Type II error more severe. A patient who believes the drug works at least 75 percent of the time will likely choose it, with his doctor, perhaps over a treatment that would have worked better.

A Type I error has a real cost too, a patient turning away from a good drug, but he still chooses some other treatment knowing its record. The Type II error misleads him about the very thing he is deciding on.

Check the direction of rejection

Why: rejecting at least 75 must mean concluding below 75, the alternative.

\[ \text{reject } H_0: p \ge 0.75 \iff \text{conclude } p < 0.75 \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 466 — Example 9.8

32. Four descriptions of an error for the cure-rate null

Error analysis

Four students described the Type II error for Example 9.8's null, that the cure rate is at least 75 percent. Only one description is right. Find what is wrong with each of the others.

Annotate

On: \( \text{Type II: keep } H_0: p \ge 0.75 \text{ when, in fact, } p < 0.75 \)

  • No truth clause. Believing the drug works is correct if it does, so this sentence does not describe an error at all.
  • Correct structure, wrong error: that rejects a true null, which is the Type I error.
  • That is the state of the world, not a decision. An error needs both the mistaken decision and the truth.
  • Correct: the null is kept while it is false, and both halves are stated in context.

Read each note against the grid. A description is only an error if its decision and its truth sit in mismatched cells: reject with a true null, or keep with a false one. Anything else is either a correct outcome or not an outcome at all.

The three wrong descriptions fail in three different ways: one has no truth, one has no decision, and one has both but in the wrong column. A complete description always has a decision, the words when in fact, and a truth that contradicts the decision.

The swapped one is the most common on tests, because the two descriptions use the same two phrases in opposite order. Anchor yourself with the null: a Type II error keeps the null, so its decision half must sound like the null.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 466 — Example 9.8

33. Trap: not rejecting read as proof

Trap

The trap

The clam test finds a sample mean of 830, below the cutoff of 841.1, and someone writes:

\[ \text{do not reject } H_0 \]

\[ \Rightarrow \text{ the clams are safe} \]

Wrong. The test failed to find strong evidence; it did not show the null is true.

The fix

A Type II error lives exactly here. If the clams truly average 850, the test lands on the do-not-reject side 36 percent of the time, as you computed. A mean of 830 is quite possible from a toxic area.

\[ \beta(850) = 0.3612 \]

The correct sentence is that there is not sufficient evidence that the mean exceeds 800. Like a jury's not guilty, it reports the weakness of the evidence, not the innocence of the clams. Only a test with high power earns any confidence in a non-rejection.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — β = probability of not rejecting a false null

34. Which error costs more

Section

Part 4

35. A judgement the arithmetic cannot make

Concept

Every example in this section ends by asking which error has the greater consequence. Notice that nothing in the probability theory answers that. Alpha and beta measure how often each error happens; neither one says how much it hurts.

The book's examples split two and two. In Examples 9.5 and 9.8 the Type II error is worse; in Examples 9.6 and 9.7 the Type I error is worse. So there is no rule of the form Type I is always worse. There is only a method.

  1. Describe both errors in context, each with its when, in fact clause.
  2. Ask what action follows each mistaken decision: what does somebody actually do?
  3. Compare the harm of the two actions: who is hurt, how badly, and can it be undone?

The third step is where people disagree, and that is fine; a good answer names the action and the harm, so that someone who disagrees can see exactly where.

\[ \text{decision} \to \text{action} \to \text{consequence} \]

Once you know which error is worse, you can act on it: make that error's probability small. If it is the Type I error, choose a small alpha. If it is the Type II error, make the power high, which usually means a larger sample. Part 5 shows how.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, pp. 464-466 — Examples 9.5-9.8

36. The book's four examples side by side

Comparison

Fill in the table from the four examples you have now seen or are about to see. The last column is the point: the worse error changes from one example to the next.

Comparison matrix

examplenull hypothesisaction after a Type I errorworse error
9.5 climbing gearthe gear is safereplaces safe gearType II
9.6 accident victimthe victim is alivestops treating a living personType I
9.7 Genetic Labsno effect on sex outcomepeople buy a useless productType I
9.8 cure ratecure rate at least 75%passes over a drug that worksType II

Look at the action column. In Examples 9.6 and 9.7 the Type I action is the harmful one: a living person left untreated, and money spent on a product that does nothing. In Examples 9.5 and 9.8 the Type I action is a cautious one, and the danger sits on the other side.

Notice too that the null column holds four quite different kinds of claim: a safety claim, a claim that someone is alive, a no-effect claim and a performance claim. What they share is the structure, an equality or default that the evidence must overturn, not any common subject.

One pattern does show. When the null says something is safe or works, keeping it wrongly exposes people to harm, so the Type II error is worse. When rejecting the null triggers a harmful act, the Type I error is worse. But the pattern is a consequence of the actions, not a rule about the labels.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, pp. 464-466 — Examples 9.5-9.8

37. Example 9.7: Genetic Labs and the sex of a baby

Worked example

A company called Genetic Labs claims it can increase the likelihood that a pregnancy results in a boy. Statisticians test the claim with the null that Genetic Labs has no effect on sex outcome. Describe both errors and say which is worse.

\[ H_0: \text{no effect} \quad (p = 0.5) \qquad H_a: \text{raises the chance of a boy} \quad (p > 0.5) \]

The book states the null in words; the symbols beside it are the deck's, with the natural chance of a boy taken as one half for simplicity. It is really about 0.51, which would change none of the reasoning.

Type I: reject a true null

Why: credit the product with an effect it does not have.

\[ \text{believes it works} \mid \text{in fact no effect} \]

Type II: keep a false null

Why: say the product does nothing when it does work.

\[ \text{believes no effect} \mid \text{in fact it works} \]

Follow each error to its action

Why: who acts on the conclusion, and how?

\[ \text{Type I} \to \text{people pay for a useless product} \]

The book judges the Type I error worse, since people would use the Genetic Labs product in hopes of increasing the chance of a boy. A published finding that it works would send couples to spend money, and hope, on nothing.

A Type II error leaves a working product unrecognised. That costs the company sales, but nobody is misled into paying for a false promise.

Check where the worse error lives

Why: the harmful action follows a rejection, so the worse error must be the Type I error.

\[ \text{harm follows reject } H_0 \Rightarrow \text{Type I worse} \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 465 — Example 9.7

38. Try It 9.7: red tide and the harvesting ban

Worked example

Massachusetts monitors toxin in clams and bans harvesting in an area when the mean level exceeds 800 micrograms per kilogram of clam meat. Describe both errors and state which has the greater consequence.

\[ H_0: \mu \le 800 \qquad H_a: \mu > 800 \]

The null is that the toxin is within the safe limit; rejecting it means concluding the clams are over the limit, which triggers or keeps the ban. This is the same test you made numerical in Part 2.

Type I: reject a true null

Why: declare the area toxic when the clams are actually safe.

\[ \text{thinks } \mu > 800 \mid \text{in fact } \mu \le 800 \]

Type II: keep a false null

Why: declare the area safe when the clams are actually toxic.

\[ \text{thinks } \mu \le 800 \mid \text{in fact } \mu > 800 \]

Follow each error to its action

Why: the ban is the action.

\[ \text{Type I} \to \text{needless ban}; \quad \text{Type II} \to \text{toxic clams sold} \]

A needless ban costs the clam diggers their income for a while. A missed toxic area lets people eat clams carrying a paralysis-inducing toxin. The Type II error has the greater consequence.

And the earlier calculation showed how big that risk is with 16 samples: about a 36 percent chance of missing an area averaging 850. That number is exactly what this judgement tells you to reduce.

Check with the numbers from Part 2

Why: the worse error is the one with the larger probability here, which is what a regulator should fix.

\[ \alpha = 0.05, \quad \beta(850) = 0.3612 \quad \Rightarrow \text{ raise the power} \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 465 — Try It 9.7

39. Which error is worse here?

Sorting

Six new situations, each with its null hypothesis. For each one, decide which error has the greater consequence. Work out the action after each error before you decide.

Sort into buckets

Sort each situation by the error with the greater consequence.

Type I error is worse
Criminal trial. H₀: the defendant is innocent.; Expensive new surgery with risky side effects. H₀: it is no better than the current one.; Email spam filter. H₀: this message is legitimate (a job offer, say).
Type II error is worse
Smoke alarm. H₀: there is no fire.; Airport baggage scan. H₀: the bag holds nothing dangerous.; Bridge inspection. H₀: the bridge is sound.
t1
Convicting an innocent person, putting patients through a risky surgery that does not help, and binning a genuine job offer are all harms caused by rejecting a true null.
t2
A silent alarm in a fire, a dangerous bag waved through and a failing bridge left open are all harms caused by keeping a false null, and each can cost lives.

The smoke alarm, the baggage scan and the bridge share a structure: the null says all is well, and keeping it wrongly means a real danger goes unnoticed. The trial, the surgery and the spam filter share the opposite one: the harmful act follows a rejection.

Notice that in every case you had to imagine a concrete scene: the fire, the bag on the plane, the patient after surgery. That is the skill being tested. The probability theory names the two errors; only the context says what they cost.

The spam filter is arguable, which is the point of the method. If you sorted it the other way, you were probably thinking of a phishing email getting through. That is a Type II error for this null, and it is a real harm too. A good answer says which action you weighed.

40. Trap: a rule about the labels

Trap

The trap

After a few examples, a student writes a rule:

\[ \text{the Type II error} \]

\[ \text{is always the worse one} \]

Wrong. The labels depend on how the null is worded; the harm does not.

The fix

Examples 9.6 and 9.7 already break the rule. Worse, the same situation can be written with either null. Restate Navah's test with the null that her gear is not safe:

\[ H_0: \text{not safe} \]

\[ \text{Type I: thinks safe} \mid \text{not safe} \]

Now climbing on unsafe gear is a Type I error. The harm never moved; only the name did. Judge the action, then read off which label it carries under the null you were given.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, pp. 464-465 — Examples 9.5-9.7

41. Flip the null and watch the labels swap

Counterexample

The previous trap claimed that rewording the null swaps the error labels. Test that on Example 9.8, the cure-rate drug, by writing the null the other way round.

Discussion prompt

Suppose the null is instead that the cure rate is less than 75 percent, with the alternative that it is at least 75 percent. Describe the Type I and Type II errors, and say which is now the more serious.

So the severity followed the action, relying on a weaker drug, and the label followed the wording. Neither label is intrinsically worse.

There is one reason the book's nulls are written as they are: the null holds the equality, at least 75 percent, as Section 9.1 requires. That convention fixes the labels for any given question, so on a test, describe the errors for the null you are given, not one you would prefer.

\[ H_0 \text{ always contains the equality: } =, \ \le, \text{ or } \ge \]

42. Smoke alarms are tuned on purpose

Real world

A smoke alarm runs a hypothesis test every few seconds, with the null that there is no fire. Its manufacturer must pick a threshold for how much smoke triggers it, which is choosing a cutoff, and so choosing between the two errors.

Discussion prompt

Most kitchen smoke alarms go off at burnt toast. Explain, in terms of Type I and Type II errors, why manufacturers set them this sensitively, and what would happen if they were made less sensitive.

Notice how the whole section appears in one household device: the two errors, the consequence judgement, the trade-off when only the threshold moves, and the escape route of better data. The same reasoning sets spam filters, medical screening tests and fraud alerts on a credit card.

In each case the designer chooses alpha deliberately large or small depending on which error hurts more, exactly the judgement you practised on the book's four examples.

43. Power, and the trade-off

Section

Part 5

44. Power: the chance of catching a false null

Concept

The fourth cell of the grid has a name of its own. When the null is false and the test rejects it, the test has done what it was built to do: it has detected a real effect. The probability of that outcome is the power of the test.

power of the test — The probability of rejecting the null hypothesis when it is false: one minus beta. Ideally it is close to one.

\[ \text{power} = 1 - \beta = P(\text{reject } H_0 \mid H_0 \text{ false}) \]

Power and beta are partners in the right-hand column of the grid. If a test misses a particular effect 36 percent of the time, it catches it the other 64 percent of the time.

Because beta depends on how false the null is, so does power. A test can have high power against a big effect and low power against a small one, which is why a power figure is always attached to a particular alternative.

The book adds one practical fact: increasing the sample size can increase the power of the test. This is the lever that does not cost you anything in false alarms, and the rest of this part shows why.

A study planned without thinking about power can be close to useless. If its power against the effect that matters is only 0.3, then even when the effect is real the study will most likely report nothing, and that empty report will be read by many as evidence of no effect.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 464 — The Power of the Test is 1 − β

45. From beta to power, and back

Worked example

Three small calculations of the kind the section's exercises ask for. In each, keep track of which column of the grid the probability belongs to.

A test has beta 0.35 against an alternative

Why: power is the rest of the same column.

\[ \text{power} = 1 - 0.35 = 0.65 \]

In context: if that alternative is true, the test detects it 65 times in 100 and misses it 35 times. Now go the other way, from a power figure to the chance of a miss.

A study is designed with power 0.90

Why: the chance of a miss is what is left.

\[ \beta = 1 - 0.90 = 0.10 \]

The clam test at a true mean of 875

Why: from the table in Part 3.

\[ \beta = 0.0877 \Rightarrow \text{power} = 1 - 0.0877 = 0.9123 \]

What you cannot do is get power from alpha. A test at alpha 0.05 has power 0.95 only by coincidence: alpha lives in the null-true column and power in the null-false column, so knowing one tells you nothing about the other without more information.

Check that each pair fills its column

Why: beta and power must add to one, and alpha is not involved.

\[ 0.35 + 0.65 = 1, \quad 0.10 + 0.90 = 1, \quad 0.0877 + 0.9123 = 1 \quad \checkmark \]

46. Turning the dial: alpha against beta

Picture it

With the sample size fixed at 16, the only thing you can change in the clam test is the cutoff. Every cutoff gives an alpha and, for a true mean of 850, a beta. Plot both against the cutoff and the trade-off appears as two curves.

Figure (svg): Two curves plotted against the cutoff c from 780 to 900. The red curve, alpha, starts near 0.79 at 780 and falls toward zero. The blue curve, beta at a true mean of 850, starts near zero and rises toward 1. They cross at c equals 825, where both equal 0.159. Dots mark c equals 841.1, where alpha is 0.05 and beta 0.36, and c equals 858.2, where alpha is 0.01 and beta 0.63.

Sliding the cutoff right lowers one curve and raises the other. With the sample size fixed, no cutoff makes both errors small: you can only choose where on these two curves to sit.

Moving the cutoff right, demanding more evidence before a ban, slides you down the red curve and up the blue one. At 841.1 you have alpha 0.05 and beta 0.36. At 858.2 you have alpha 0.01 and beta 0.63.

The two curves cross at 825, exactly halfway between 800 and 850, where both errors equal 0.159. There is no cutoff where both are small; you only get to choose your point on the curves, and the consequence judgement from Part 4 tells you which way to lean.

\[ c = 825: \quad \alpha = \beta = P(Z > 1) = 0.1587 \]

47. The trade-off: lowering alpha to 0.01

Worked example

Shellfish farmers complain that a 5 percent false-ban rate is too high, and the regulator lowers alpha to 0.01 while keeping 16 samples. Find the new cutoff and the new beta for an area averaging 850.

New z

Why: now only 1 percent above it.

\[ z_{0.99} = 2.3263 \]

New cutoff

Why: 800 plus 2.3263 standard errors of 25.

\[ c = 800 + 2.3263(25) = 800 + 58.16 = 858.2 \]

The cutoff has moved up by about 17 micrograms per kilogram. A truly toxic area averaging 850 now sits below the cutoff, so most of its samples will not trigger a ban.

Standardise under the truth

Why: the cutoff is now above 850.

\[ z = \frac{858.16 - 850}{25} = \frac{8.16}{25} = 0.326 \]

New beta

Why: the area left of the cutoff under the 850 curve.

\[ \beta = P(Z < 0.326) = 0.6279 \]

Figure (svg): Two panels, each showing the null curve centred at 800 and the true curve centred at 850, with the same width. Left: cutoff 841.1, a small red alpha tail of 0.05, and blue beta area 0.36. Right: cutoff moved right to 858.2, a red sliver of 0.01, and a much larger blue beta area of 0.63.

Moving the cutoff right to protect against false alarms shrinks the red tail and grows the blue region. The two curves did not change; only the line moved.

In context: cutting false bans from 5 percent to 1 percent raises the chance of missing an area averaging 850 from 36 percent to 63 percent. Given the Type II error is the dangerous one here, that is a poor bargain.

Nothing about the clams changed between the two calculations, and nothing about the sample. The regulator simply moved the line, and the line cannot move away from one error without moving towards the other.

Check the direction of the change

Why: smaller alpha must mean a higher cutoff and a larger beta.

\[ \alpha: 0.05 \to 0.01, \quad c: 841.1 \to 858.2, \quad \beta: 0.361 \to 0.628 \quad \checkmark \]

48. Why a bigger sample beats the trade-off

Intuition

Moving the cutoff only redistributes the overlap between the two curves. To shrink both errors you have to shrink the overlap itself, and the overlap is controlled by how wide the curves are.

The width of each curve is the standard error, sigma over the square root of n. Quadruple the sample and the standard error halves, while the centres, 800 and 850, stay put.

\[ \frac{100}{\sqrt{16}} = 25 \quad \to \quad \frac{100}{\sqrt{64}} = 12.5 \]

Measured in standard errors, the gap between the null and the truth doubles, from 2 to 4. Two curves that far apart barely overlap, so a cutoff can sit between them with almost nothing of either curve on the wrong side.

\[ \frac{850 - 800}{25} = 2 \quad \to \quad \frac{850 - 800}{12.5} = 4 \]

That is the book's remark that increasing the sample size can increase the power, made visible. The sample size is the one lever that improves both errors at once; its cost is not statistical but practical: more clams to collect, more patients to enrol, more money.

49. The fix: 64 samples instead of 16

Worked example

The regulator keeps alpha at 0.05 but collects 64 samples per area. Find the new cutoff and beta for a true mean of 850, and compare with the 16-sample test.

New standard error

Why: the square root of 64 is 8.

\[ \sigma_{\bar{x}} = \frac{100}{\sqrt{64}} = 12.5 \]

New cutoff

Why: still 1.6449 standard errors above 800, but the step is shorter.

\[ c = 800 + 1.6449(12.5) = 800 + 20.56 = 820.6 \]

Standardise under the truth

Why: 850 is now far beyond the cutoff.

\[ z = \frac{820.56 - 850}{12.5} = \frac{-29.44}{12.5} = -2.355 \]

Beta and power

Why: the left tail is tiny.

\[ \beta = P(Z < -2.355) = 0.0093, \quad \text{power} = 0.9907 \]

Figure (svg): Two tall narrow bell curves, centred at 800 and 850, each with standard error 12.5 for samples of 64; the wider dashed grey curves for samples of 16 sit behind them. The cutoff is now 820.6. The red alpha tail is still 0.05, but the blue beta area of the 850 curve left of the cutoff is a sliver of 0.009.

Four times the sample halves the standard error, so the two curves pull apart and barely overlap. Alpha stays at 0.05 by design, and beta collapses from 0.36 to 0.009.

In context: with 64 samples per area, a test that still bans a safe area only 5 percent of the time now catches an area averaging 850 more than 99 times in 100. Alpha did not move, and beta fell from 0.36 to under 0.01.

Notice the coincidence in the numbers: the z for 850 at n equal to 64 is the same as for 900 at n equal to 16. Doubling the gap in standard errors has the same effect whether it comes from a bigger effect or a bigger sample.

Why did the cutoff move down, closer to 800? Because alpha is still 0.05, and with a narrower null curve its top 5 percent begins closer to the centre. The bigger sample lets the test react to a smaller excess without raising the false-alarm rate.

Check alpha is unchanged

Why: standardise the new cutoff under the null.

\[ P\left(Z > \frac{820.56 - 800}{12.5}\right) = P(Z > 1.645) = 0.05 \quad \checkmark \]

50. Watch the power curve as n grows

Tweak it

The slider draws the power of the clam test, at alpha 0.05, against the true mean toxin level. Its height at any true mean is the chance the test bans that area. Move the sample size and watch the curve.

Parameter explorer

Start at n equal to 4. Read the power at a true mean of 850. Then move n to 16 and to 64 and read it again. Where does every curve start, at a true mean of 800, whatever n is?

\[ \text{power}(\mu) = P\left(Z > 1.645 - \frac{\mu - 800}{100/\sqrt{{n}}}\right) \]

  • n — from 1 to 100: sample size n

Two features never change as you drag. The curve always passes through 0.05 at 800, because at that point the null is exactly true and a rejection is a false alarm, which happens at rate alpha by design.

And the curve always rises from left to right: the further the truth from the limit, the easier it is to detect. What the sample size controls is how steeply it rises, which is how close to the limit a toxic area can be and still be caught reliably.

\[ \text{power}(800) = \alpha = 0.05 \text{ for every } n \]

51. Three sample sizes, three power curves

Picture it

Here are three of the curves from the slider drawn together, for 4, 16 and 64 samples per area. Each curve is a whole table of power values, one for every possible true mean.

Figure (svg): Three S-shaped power curves plotted against the true mean toxin level from 780 to 920, for samples of 4, 16 and 64. All three pass through 0.05 at a true mean of 800. The curve for 64 rises steeply and is near 1 by 850; the curve for 16 reaches 0.64 at 850; the curve for 4 rises slowly, only 0.26 at 850. A dashed line marks 0.05.

Every curve starts at 0.05 where the null is exactly true, because rejecting there is a Type I error. Further right the truth is further from 800, and power climbs; a bigger sample makes it climb sooner.

Read the curves at a true mean of 850. With 4 samples the test catches it about a quarter of the time; with 16, about two thirds; with 64, essentially always. Read them at 820, just over the limit, and even 64 samples catch it less than half the time.

So a power curve answers the question a regulator, a drug company or a researcher actually cares about: if the truth is this bad, how likely am I to find out? That is the question behind every sample-size calculation.

And notice the left end. To the left of 800 the null is true, so a rejection there is a Type I error, and every curve is at or below 0.05. A power curve shows both errors at once.

52. How many samples for power 0.90?

Worked example

The regulator wants a 90 percent chance of catching an area averaging 850, with alpha 0.05. How many samples per area are needed? The test rejects above 800 plus 1.6449 standard errors; you need that cutoff to sit 1.2816 standard errors below 850.

\[ z_{0.95} = 1.6449, \quad z_{0.90} = 1.2816 \]

Why 1.2816? A power of 0.90 means 90 percent of the 850 curve must lie right of the cutoff, so the cutoff sits at that curve's 10th percentile, 1.2816 standard errors below its centre.

Require the gap to cover both

Why: the distance from 800 to 850 must equal 1.6449 plus 1.2816 standard errors.

\[ 850 - 800 = (1.6449 + 1.2816)\frac{100}{\sqrt{n}} \]

Solve for the square root of n

Why: multiply across and divide by 50.

\[ \sqrt{n} = \frac{2.9265(100)}{50} = 5.853 \]

Square

Why: and round up, since power must be at least 0.90.

\[ n = 5.853^{2} = 34.26 \Rightarrow n = 35 \]

Figure (svg): A rising curve of the power to detect a true mean of 850, plotted against sample size n from 1 to 80. It starts near 0.13 and climbs past 0.64 at n equals 16, crossing a dashed line at 0.90 between 34 and 35, and is above 0.99 by 64. A dot marks n equals 35 with power 0.905.

Power rises steeply at first and then flattens. Going from 16 to 35 samples lifts it from 0.64 to 0.90; after that, each extra sample buys less.

In context: averaging 35 samples per area gives the regulator at least a 90 percent chance of banning harvesting where clams average 850 micrograms per kilogram, while keeping false bans at 5 percent.

Check the power at n = 35

Why: compute the cutoff and the 850 curve's area beyond it.

\[ c = 800 + 1.6449\tfrac{100}{\sqrt{35}} = 827.8, \quad P\left(Z > \tfrac{827.8 - 850}{16.90}\right) = 0.9054 \quad \checkmark \]

53. Rank four designs by power

Ranking

Four designs for the clam test, all judged by their power against an area averaging 850. Rank them from most powerful to least, using what you know about how alpha and the sample size move the curves.

Put in order

Rank from highest power to lowest power at a true mean of 850.

  1. n = 64, α = 0.05
  2. n = 64, α = 0.01
  3. n = 16, α = 0.05
  4. n = 16, α = 0.01

Why: A bigger sample raises power a great deal and a smaller alpha lowers it. With 64 samples the powers are 0.991 at the 0.05 level and 0.953 at the 0.01 level; with 16 samples they are 0.639 and 0.372.

designcutoff cpower at μ = 850
n = 64, α = 0.05820.60.991
n = 64, α = 0.01829.10.953
n = 16, α = 0.05841.10.639
n = 16, α = 0.01858.20.372

The striking comparison is the middle two. The design with 64 samples at the strict 0.01 level beats the design with 16 samples at the looser 0.05 level by a wide margin. A larger sample buys both a smaller alpha and a higher power.

54. Genetic Labs as an exact test: 20 births

Worked example

Make Example 9.7 numerical with a design of our own. Twenty couples use the product, and the statisticians reject the no-effect null if 15 or more of the 20 babies are boys. Find alpha, then beta if the product really raises the chance of a boy to 0.7.

\[ X = \text{boys in 20 births}, \quad H_0: p = 0.5, \quad \text{reject if } X \ge 15 \]

Alpha is a right tail of the null binomial

Why: add the probabilities of 15 through 20 boys when p is one half.

\[ \alpha = P(X \ge 15 \mid p = 0.5) = \sum_{k=15}^{20}\binom{20}{k}(0.5)^{20} \]

Add the counts

Why: the binomial coefficients from 15 to 20 boys.

\[ \alpha = \frac{15504 + 4845 + 1140 + 190 + 20 + 1}{1048576} = \frac{21700}{1048576} = 0.0207 \]

Figure (svg): Bars for the number of boys in 20 births when the chance of a boy is 0.5, symmetric about 10 with the tallest bar about 0.176. The bars for 15 to 20 boys are red, the rejection region; together they hold 0.0207 of the probability.

With no effect at all, 15 or more boys out of 20 happens only about 2 times in 100. That small red area is the chance of crediting the product with an effect it does not have.

So with no effect at all, this study falsely credits the product about 2 times in 100. That is a small alpha, as it should be, given that the Type I error is the worse one here.

Beta under p = 0.7

Why: the product works, but the count still falls below 15.

\[ \beta = P(X \le 14 \mid p = 0.7) = 0.5836 \]

Power

Why: one minus beta.

\[ 1 - 0.5836 = 0.4164 \]

Figure (svg): Bars for the number of boys in 20 births when the chance of a boy is 0.7, peaking at 14 with height about 0.19. The bars from 0 to 14 boys are blue, the do-not-reject region, holding 0.5836 of the probability; the bars from 15 to 20 are green, holding the power, 0.4164.

Now the product really raises the chance of a boy to 0.7, and still more than half of all studies of 20 births fail to reach 15 boys. A real effect is missed 58 times in 100.

In context: even if the product really raises the chance of a boy from one half to 0.7, a 20-birth study detects it only about 42 percent of the time. Small studies of real effects usually come up empty.

Compare the two pictures. The cutoff at 15 is the same line in both; only the distribution underneath changed. That is the clam test's two-curve picture again, drawn with bars because a count of boys can only be a whole number.

Check alpha against the bar picture

Why: the red bars must sum to the computed alpha.

\[ 0.01479 + 0.00462 + 0.00109 + 0.00018 + 0.00002 + 0.000001 = 0.0207 \quad \checkmark \]

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, p. 465 — Example 9.7 (the design is the deck's own)

55. Can the Genetic Labs test hit alpha 0.05 exactly?

Edge cases

Suppose the statisticians want alpha as close to 0.05 as possible. With a count of boys, the cutoff must be a whole number, so only a few values of alpha are available. Push the cutoff one step each way.

reject if boys ≥α = P(reject | p = 0.5)β at p = 0.7power at p = 0.7
140.05770.39200.6080
150.02070.58360.4164
160.00590.76250.2375

Discussion prompt

No cutoff gives alpha of exactly 0.05. Which cutoff would you choose if you insist that alpha be at most 0.05, and what does that choice cost in power? What could the statisticians change to get both a small alpha and a high power?

The table is the trade-off from the clam test in discrete form: every step of the cutoff that lowers alpha raises beta. The only difference is that with a count you cannot move the dial smoothly, so the alpha you get is usually smaller than the alpha you asked for.

\[ \alpha \downarrow \Rightarrow \beta \uparrow \quad \text{(fixed } n = 20\text{)} \]

56. Why studies report alpha but rarely beta

Concept

Almost every published test states its significance level. Far fewer state beta or the power. The reason is built into the definitions you have seen.

Alpha is computed under the null, which names one exact value, 800 micrograms or a probability of one half. So alpha is a single number, and the researcher chooses it.

\[ \alpha = P(\text{reject} \mid \mu = 800) \quad \text{one number, chosen} \]

Beta is computed under the alternative, which is a whole range of values. To give a number you must pick one: the effect size you care about. Different readers care about different effect sizes, and the true effect is unknown, so there is no single beta to report.

\[ \beta(\mu) \quad \text{a different number for every } \mu > 800 \]

Good studies solve this by stating, in advance, the smallest effect worth detecting and the power they designed for, as in the sample-size calculation: 35 samples for power 0.90 at 850. When a study reports a non-significant result without that information, you cannot tell whether the effect is absent or the study was simply too small to see it.

57. What fraction of rejections are wrong?

Worked example

A lab runs 1000 tests a year at alpha 0.05. Suppose that in 100 of them the null is really false, and that against those effects the tests have power 0.80. Of all the rejections, what fraction are Type I errors?

Split by the truth

Why: the column of the grid comes first.

\[ 1000 = 900 \text{ true nulls} + 100 \text{ false nulls} \]

Type I errors among the true nulls

Why: alpha of them are rejected.

\[ 900 \times 0.05 = 45 \]

Correct rejections among the false nulls

Why: the power of them are rejected.

\[ 100 \times 0.80 = 80 \]

All rejections

Why: add the two rejecting branches.

\[ 45 + 80 = 125 \]

Fraction of rejections that are wrong

Why: the Type I errors out of all rejections.

\[ \frac{45}{125} = 0.36 \]

Figure (svg): A tree diagram. A box of 1000 tests run at alpha 0.05 splits into 900 tests where H0 is true and 100 where it is false. The 900 split into 45 rejected, a red Type I box, and 855 kept, correct. The 100 split into 80 rejected, a green correct box labelled power 0.80, and 20 kept, Type II. A line beneath reads: rejections 45 plus 80 equals 125, and 45 of them are wrong.

Alpha is the red share of the top branch only: 45 out of 900. Among all the rejections, the wrong ones are 45 out of 125, which is a different fraction.

In context: in this lab, more than a third of the significant results are false alarms, even though every test was run at the 5 percent level. The 0.05 describes the top branch only; the 0.36 depends also on how many nulls are false and on the power.

Check that every test is accounted for

Why: the four branch counts must add back to 1000.

\[ 45 + 855 + 80 + 20 = 1000 \quad \checkmark \]

58. Trap: alpha as the chance your result is wrong

Trap

The trap

A test at the 0.05 level rejects the null, and the write-up says:

\[ \text{there is a 5\% chance} \]

\[ \text{this conclusion is wrong} \]

Wrong. Alpha is a probability given that the null is true, not given that you rejected.

The fix

Alpha answers: if the null is true, how often does the test reject? The write-up needs: given a rejection, how often is the null true? Those are different conditional probabilities.

\[ \alpha = P(\text{reject} \mid H_0) \]

\[ \ne P(H_0 \mid \text{reject}) \]

The tree showed they can differ wildly: alpha 0.05, yet 36 percent of rejections wrong. A correct sentence is: if the null were true, a result this extreme would occur in fewer than 5 percent of samples.

59. Putting it together

Section

Part 6

60. Pattern: describe, judge, then design

Pattern

Every question in this section, from the book's word problems to the power calculations, runs through the same four moves. The first two are about language, the last two about numbers.

  1. Write the null in words, and say what rejecting it means in context.
  2. Describe both errors as a decision plus a when, in fact clause: Type I rejects a true null, Type II keeps a false one.
  3. Judge the consequence by following each decision to the action it triggers and the harm if it is wrong.
  4. Design against the worse error: a small alpha if it is Type I; high power, usually from a larger sample, if it is Type II.

\[ \alpha = P(\text{reject} \mid H_0 \text{ true}), \quad \beta = P(\text{keep} \mid H_0 \text{ false}), \quad \text{power} = 1 - \beta \]

Three facts carry the numerical side. Alpha is chosen; beta depends on the true value. With the sample size fixed, lowering one raises the other. A larger sample lowers both.

And one fact carries the interpretive side: alpha is a probability given that the null is true. It is never the probability that the null is true, nor the probability that a particular conclusion is wrong.

61. Check: describing a Type II error

Check

A new context. A factory tests whether a batch of bolts meets its strength specification, with the null hypothesis that the batch meets it.

Check your understanding

For the null hypothesis that the batch of bolts meets the strength specification, which statement describes a Type II error?

  • A. The batch is accepted as meeting spec when, in fact, it does not meet spec. (correct)
  • B. The batch is rejected as below spec when, in fact, it meets spec.
  • C. The batch does not meet spec.
  • D. The batch is accepted as meeting spec when, in fact, it meets spec.

Answer: A

Why: A Type II error keeps a null that is false. Here the null is that the batch meets spec, so keeping it, accepting the batch, while the batch is actually substandard is the Type II error, and weak bolts go out to customers.

Why B tempts people
That rejects a null that is true, a good batch thrown out, which is the Type I error.
Why C tempts people
That states the truth without any decision, so it is not an error at all; an error needs both halves.
Why D tempts people
The decision matches the truth, so this is a correct outcome, not an error.

\[ H_0: \text{meets spec} \quad \text{Type II: keep } H_0 \mid H_0 \text{ false} \]

Each wrong option fails one of the tests from this lesson: the wrong column, no decision, or no mismatch. In this context the Type II error is also the dangerous one, since substandard bolts end up in buildings and machines.

62. Check: power with 36 samples

Check

Return to the clam test, alpha 0.05 and standard deviation 100, but with 36 samples per area. The standard error is 100 over 6, which is 16.67, and the cutoff is 827.4.

Check your understanding

With 36 samples, what is the power of the clam test against an area whose clams truly average 850?

  • A. about 0.912 (correct)
  • B. about 0.639
  • C. about 0.088
  • D. about 0.950

Answer: A

Why: The cutoff 827.4 is 1.355 standard errors of 16.67 below 850, so the area of the 850 curve beyond the cutoff is the area right of z equal to minus 1.355, which is 0.912.

Why B tempts people
That is the power with 16 samples, where the standard error is 25; with 36 samples the curves are narrower and the power is higher.
Why C tempts people
That is beta, the chance of missing the area, not the power, which is one minus beta.
Why D tempts people
That is one minus alpha, a probability in the null-true column; it says nothing about detecting a mean of 850.

\[ z = \frac{827.41 - 850}{16.667} = -1.355, \quad \text{power} = P(Z > -1.355) = 0.9123 \]

In context: with 36 samples per area, the regulator bans harvesting in an area averaging 850 about 91 percent of the time. Note the coincidence: 36 samples at 850 gives the same power as 16 samples at 875, because both put the truth 3 standard errors from 800.

63. Check: what alpha 0.05 means

Check

The last check is about the sentence, not the arithmetic. Read every option for what it claims alpha is a probability of.

Check your understanding

A researcher runs a test at significance level 0.05. Which statement about the 0.05 is correct?

  • A. If the null hypothesis is true, the test will wrongly reject it 5% of the time in the long run. (correct)
  • B. If the test rejects the null hypothesis, there is a 5% chance the null hypothesis is true.
  • C. There is a 95% chance the test detects a real effect.
  • D. There is a 5% chance the null hypothesis is true.

Answer: A

Why: Alpha is the probability of rejecting the null given that the null is true: over many tests of true nulls, the rejection rate is five percent. It is a property of the procedure under the null, nothing more.

Why B tempts people
That reverses the condition. The chance a rejection is wrong depends on how many nulls are false and on the power; in the lab example it was 36 percent.
Why C tempts people
That is a claim about power, which depends on the sample size and the effect, and is not one minus alpha.
Why D tempts people
The null is either true or false; alpha is not a probability about the hypothesis itself, but about what the test does when the null is true.

All three wrong options are things people genuinely write in reports. The second is the most common and the most misleading, because it sounds like what you want to know. The tree diagram is the antidote: alpha is the red share of the true-null branch only.

64. Explain why you cannot have it all

Explain it to yourself

This is the idea that ties the section together. Explaining it in your own words is the best test of whether the pictures have landed.

Discussion prompt

With the sample size fixed, why can a test not make both alpha and beta as small as you like? Use the two-curve picture of the clam test, and say what does reduce both.

A strong answer names three things: there is one cutoff and two curves; the errors are areas on opposite sides of the cutoff under different curves; the sample size controls the overlap. The numbers illustrate the argument rather than replace it.

If your explanation said alpha and beta add to one, look again at the grid: they are in different columns, so their sum means nothing.

65. Exit ticket: the battery claim

Exit ticket

A manufacturer claims its batteries last at least 500 hours on average. A consumer group tests 25 batteries, with a known standard deviation of 40 hours, at alpha 0.05, rejecting the claim if the sample mean is below a cutoff.

Discussion prompt

(a) Describe the Type I and Type II errors in context. (b) Find the cutoff. (c) If the batteries really average 480 hours, find beta and the power. (d) Which error matters more to a consumer group, and what would you change to reduce it?

Part (d) is where the whole lesson meets: the consumer group names the worse error from its own consequences, then reaches for the one lever that reduces it without raising the other error, the sample size.

The marks are in the language as much as the numbers. The cutoff is below 500 because the alternative points down, so beta is the area of the 480 curve above the cutoff, the batteries that look fine although they are not.

Check yourself: the power of 0.80 means that a group testing 25 batteries catches a 20-hour shortfall four times in five. And the Type I error is the one the manufacturer worries about, which is a reminder that the worse error depends on whose consequences you weigh.

66. Recap

Recap

A test decides from a sample without seeing the truth, so its decision can be wrong in two ways. Cross the two decisions with the two truths and there are four outcomes.

H₀ trueH₀ false
do not reject H₀correct (1 − α)Type II error (β)
reject H₀Type I error (α)correct: power (1 − β)

Alpha is chosen in advance, and is the false-alarm rate when the null is true. Beta depends on how false the null is, and is the miss rate for that particular truth. Each error is described as a decision plus a when, in fact clause in the context of the stated null.

Which error is worse is a judgement about consequences: follow each decision to its action and weigh the harm. The book's four examples split two and two, so the label never decides on its own.

With the sample size fixed, lowering alpha raises beta. A larger sample narrows the sampling distributions and lowers both, which is why it raises the power. Introductory Business Statistics treats the same outcomes in business contexts, if you want more practice.

OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2, pp. 464-466 — the whole section

OpenStax Introductory Business Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors §9.2 Outcomes and the Type I and Type II Errors — the same ideas in business contexts

Sources

  1. OpenStax Introductory Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors — Illowsky & Dean, OpenStax / Rice University, CC BY 4.0, pp. 463-467
  2. OpenStax Introductory Business Statistics 2e, §9.2 Outcomes and the Type I and Type II Errors — Illowsky & Dean, OpenStax / Rice University, CC BY 4.0

Want this taught 1-on-1? Alexander tutors Statistics — $55/session, free consultation.

Book on Wyzant · Text (657) 465-8108