Statistical Inference · Statistical Inference
What a p-value is
In Statistical Inference because p is the chance of a result this weird if H0 were true — not the chance H0 is true.
The last lesson counted sheets. A p-value is that count turned into a chance: if H0 were the world, how often would a sample this weird, or weirder, appear? For 5 of 8 under a coin, that is 186/256. For 8 of 8, that is 2/256. The number is about the data under H0. It is not a vote on whether H0 is true.
- Statistical Inference
- Medium level
- 5 concepts
1Weird if H0 were the world
Fix H0 as true for a moment — the coin story, come-back chance one half. Then ask: in that world, how often do I see a come-back count as far from 4 as the one I got, or farther? That chance is the p-value.
The phrase to keep is this weird if H0 were true. Swap the nouns and you get the famous trap: 'the probability H0 is true'. That is a different question, and this number does not answer it. p starts by assuming the dull story, then prices the sheet.
Figure. p assumes H0 is a coin, looks at 5 of 8, and asks how often that world makes something this weird.
How p is read
- Assume H0Pretend the coin story is the world.
- Name the data5 of 8, or 8 of 8, or whatever you saw.
- pChance of a result this far out, in that world.
Coding lab. Name the ingredients runs in the app, with checks on your output.
What does a p-value actually calculate in a statistical test?
- The probability of observing data at least as extreme as the sample, assuming the null hypothesis is true
- The probability that the null hypothesis is true given the collected data
- The probability that the research hypothesis is completely incorrect
- The proportion of population members who agree with the sample mean
The p-value is P(Data as or more extreme | H0 is true), NOT P(H0 | Data).
2p for 5 of 8
Under a fair coin there are 256 equally likely 8-row sheets. 186 of them are at least one step from 4 yes — the 5, 6, 7, 8 counts and the matching 3, 2, 1, 0 tail. So p = 186 / 256.
186 / 256 = 0.727. That is a large p. Large means: H0's world makes this kind of sheet all the time. It matches fail-to-reject. It is not a 73 percent chance the coin story is true.
Figure. Under a fair coin there are 256 equally likely 8-row sheets. 186 of them are at least one step from 4 yes — the 5, 6, 7, 8 counts and the matching 3, 2, 1, 0 tail. 186 / 256 = 0.727. Large p means H0's world makes this kind of sheet all the time. It is not a 73 percent chance the coin story is true.
186 in 256
What is the two-sided p-value for 5 yes in 8 fair-coin rows?
- sheets as far as 5186
- 186 / 2560.727
Pro tip. 0.727 is 'this ordinary if H0 were true'. It is not P(H0).
Coding lab. p for 5 of 8 runs in the app, with checks on your output.
Under a fair coin (H0: p=0.5), the probability of getting at least as extreme as 5 of 8 heads is 186/256 = 0.727. What is the interpretation?
- The fair coin model is rejected with 72.7% confidence
- The customer return rate has been proven to equal 72.7%
- Observing 5 of 8 is completely ordinary under H0, yielding no evidence against the fair coin model
- The sample size n=8 is mathematically invalid for binomial probabilities
A p-value of 0.727 is very large (> 0.05), meaning this outcome is common under the null hypothesis.
3p for 8 of 8
The all-yes sheet, plus the all-no sheet, is 2 of 256. p = 2 / 256 = 0.008. That is a small p. Small means: H0's world almost never makes this sheet. The same recipe as 5 of 8; only the count of extreme sheets changed.
The p-value moved because the data moved — 8 of 8 instead of 5 of 8 — not because we changed the meaning of p. One number is ordinary under a coin. The other is rare. Both are about the sheet in a coin world.
Figure. p is 0.727 for 5 of 8 and 0.008 for 8 of 8. Same H0. Different weirdness.
2 in 256
What is the two-sided p-value for 8 yes in 8 fair-coin rows?
- all yes + all no2
- 2 / 2560.008
Pro tip. 0.008 is 'this rare if H0 were true'. Still not P(H0).
Coding lab. p for 8 of 8 runs in the app, with checks on your output.
Observing 8 out of 8 returning customers has a two-sided p-value of 2/256 = 0.0078. What does this small p-value imply?
- The null hypothesis has been verified with 99.2% probability
- This outcome is very rare under a fair coin null, providing strong evidence against H0
- The shop owner is guaranteed to have 100% retention on all future weeks
- The test has failed due to an insufficient number of non-returning customers
A small p-value (0.0078 < 0.05) means the data is surprising under H0, justifying rejection of the null.
4p is not P(H0)
p started by assuming H0. A number that assumes H0 cannot be the probability that H0 is true. That would be answering a question after using the answer as a premise. 0.727 assumed the coin, then asked how often 5-of-8-or-weirder appears.
So 0.727 is not 'the coin is probably true', and 0.008 is not 'the coin is probably false'. Both numbers are about how surprising the sheet is in a coin world. Belief about H0 is a different conversation — and not this course's p-value.
No diagram — the idea is carried by the prose, table, code block or coding lab.
| Question | Does p answer it? |
|---|---|
| This weird if H0 were true? | Yes |
| Probability H0 is true? | No |
p = 0.727 for 5 of 8. What does that number mean?
- If the coin story were true, a sheet this ordinary shows up about 73 percent of the time
- There is a 73 percent chance H0 is true
- There is a 27 percent chance Ha is true
p assumes H0 and talks about the data. It is not P(H0).
50.05 is a habit
People often call p below 0.05 'significant' and reject H0. That cut is a habit, not a law of the shop. 0.008 falls under it; 0.727 does not. The interesting fact is still the two counts: 186/256 versus 2/256.
Do not worship the cut. A p of 0.049 and a p of 0.051 are almost the same sheet. The shop's 0.727 versus 0.008 are not. Report the p, and the decision, and the story you actually tested — here, the fair-coin come-back story.
No diagram — the idea is carried by the prose, table, code block or coding lab.
| Sheet | p | Vs 0.05 |
|---|---|---|
| 5 of 8 | 0.727 | Far above — fail to reject |
| 8 of 8 | 0.008 | Below — reject |
Coding lab. Both p's runs in the app, with checks on your output.
Why is the alpha = 0.05 significance threshold described as a convention rather than a fundamental law?
- Because p-values below 0.05 always represent computational rounding errors
- Because scikit-learn strictly forbids setting alpha to any value other than 0.05
- Because 0.05 is the minimum possible standard error in binomial experiments
- It is a historical decision threshold, not an absolute boundary where truth suddenly flips
0.05 is an agreed-upon convention for controlling Type I error; reality does not jump discontinuously at p = 0.049 vs p = 0.051.
Notes
- In Statistical Inference because p is the chance of a result this weird if H0 were true — not the chance H0 is true.
- Fix H0 as true for a moment — the coin story. Then ask: in that world, how often do I see a come-back count as far from 4 as the one I got? That chance is the p-value.
- Under a fair coin there are 256 equally likely 8-row sheets. 186 of them are at least one step from 4 yes. So p = 186 / 256.
Exam traps & shortcuts
- A sample number describes these rows. A population claim reaches past them.
- Fail-to-reject is not proof. A p-value is not the chance the null is true.
Recap
p = how often H0's world makes a result this weird. 0.727 for 5 of 8; 0.008 for 8 of 8. Neither number is P(H0). 0.05 is a habit, not a law of the shop.
- Weird if H0 were the world
- Fix H0 as true for a moment — the coin story, come-back chance one half. Then ask: in that world, how often do I see a come-back count as far from 4 as the one I got, or farther? That chance is the p-value.
- p for 5 of 8
- Under a fair coin there are 256 equally likely 8-row sheets. 186 of them are at least one step from 4 yes — the 5, 6, 7, 8 counts and the matching 3, 2, 1, 0 tail. So p = 186 / 256.
- p for 8 of 8
- The all-yes sheet, plus the all-no sheet, is 2 of 256. p = 2 / 256 = 0.008. That is a small p. Small means: H0's world almost never makes this sheet. The same recipe as 5 of 8; only the count of extreme sheets changed.
- p is not P(H0)
- p started by assuming H0. A number that assumes H0 cannot be the probability that H0 is true. That would be answering a question after using the answer as a premise. 0.727 assumed the coin, then asked how often 5-of-8-or-weirder appears.
Practise What a p-value is
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 3-question practice set that ends the chapter
- 5 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device