Engineering Mathematics · Probability & Statistics
Probability Inside Evaluation Metrics
In Engineering Mathematics because accuracy, precision, recall, and calibration all summarize uncertain outcomes through probability language.
Model metrics look like dashboard numbers, but many of them are probability summaries in disguise. This topic reads confusion-matrix metrics and calibration as conditional-frequency statements.
- Engineering Mathematics
- Medium level
- 4 concepts
1Precision and recall are conditional probabilities
Precision asks: inside the predicted-positive world, how often was the model right? Recall asks: inside the actual-positive world, how often did the model catch the case?
Both use the same true-positive count, but the denominators are different. That denominator is the event being conditioned on.
Figure. Precision conditions on predicted positives; recall conditions on actual positives. The denominators are different bars.
Read the denominators
- Precision worldCondition on predicted positives.
- Recall worldCondition on actual positives.
- Accuracy worldUse all predictions as the denominator.
| Actual positive | Actual negative | Total | |
|---|---|---|---|
| Predicted positive | 30 | 10 | 40 |
| Predicted negative | 20 | 40 | 60 |
| Total | 50 | 50 | 100 |
Compute three metrics from one table
Use TP=30, FP=10, FN=20, TN=40.
- Precision30/(30+10)=30/40=0.75
- Recall30/(30+20)=30/50=0.60
- Accuracy(30+40)/100=0.70
Pro tip. The true-positive count is shared, but each metric conditions on a different denominator.
A classifier has TP=45, FP=15, FN=30, TN=110. What is precision?
- 45/(45+15)=0.75
- 45/(45+30)=0.60
- (45+110)/200=0.775
- 15/(45+15)=0.25
Precision conditions on predicted positives: true positives divided by TP plus FP.
2Accuracy ignores how confident the model was
Accuracy grades hard correct/incorrect outcomes. It does not remember whether the model said 0.51 or 0.99 before the threshold converted a score into a label.
That makes accuracy useful but incomplete. Two models can have the same accuracy while one gives honest probabilities and the other is badly overconfident.
Figure. Accuracy on this slice is 6/10 = 0.60. The model stated confidence 0.90 on every positive. Accuracy does not record the 0.30 gap between claimed probability and observed frequency.
Separate label and score
- ScoreA model outputs a probability-like number.
- ThresholdA decision rule turns the score into a hard label.
- AccuracyThe metric checks only the hard label against truth.
Same labels, different confidence
A model makes 10 positive predictions and 6 are correct. It assigned probability 0.90 to all 10 positives.
- Hard-label accuracy on this slice6/10=0.60
- Average stated confidence0.90
- Observed correctness frequency0.60
- Confidence gap0.90-0.60=0.30
Pro tip. Accuracy says 60 percent were correct; it does not by itself punish the 90 percent confidence claim.
Two models both classify 80 of 100 labels correctly. One assigns confident 0.99 scores and one assigns calibrated 0.80 scores. What does accuracy alone say?
- They tie on hard-label accuracy
- The 0.99 model is necessarily calibrated
- The 0.80 model is necessarily wrong
- Accuracy measures score honesty
Accuracy only counts correct hard labels; it does not inspect whether probability scores were honest.
3Calibration asks whether probabilities mean what they say
A calibrated probability score has a frequency interpretation: among many cases scored near 0.80, about 80 percent should be positive.
Calibration is therefore a probability-quality metric, not a hard-label metric. It checks whether confidence values can be trusted for downstream decisions.
Figure. Calibration compares predicted probability with observed frequency. A well-calibrated model stays near the diagonal.
Check one score bin
- GroupCollect cases with similar predicted probabilities.
- ObserveCompute the actual positive frequency in that group.
- CompareSubtract observed frequency from stated probability.
| score bin | cases | positives | observed frequency |
|---|---|---|---|
| 0.80 | 20 | 16 | 0.80 |
| 0.90 | 10 | 6 | 0.60 |
One calibrated bin and one overconfident bin
A 0.80-score bin has 20 cases and 16 positives. A 0.90-score bin has 10 cases and 6 positives.
- 0.80 bin frequency16/20=0.80
- 0.80 bin gap0.80-0.80=0.00
- 0.90 bin frequency6/10=0.60
- 0.90 bin gap0.90-0.60=0.30
Pro tip. The 0.80 bin is calibrated in this tiny check; the 0.90 bin is overconfident by 0.30.
A score bin near 0.70 contains 50 cases and 35 positives. What is the observed frequency?
- 0.70
- 0.35
- 0.50
- 0.15
35/50=0.70, matching the stated score in this tiny bin.
4Metric choice decides which probability question matters
Choosing a metric is choosing a question about events. Accuracy asks about overall correctness, precision asks about reliability among predicted positives, recall asks about coverage among actual positives, and calibration asks whether scores can be trusted.
Engineering decisions should name the risk first. A medical screen, fraud queue, and recommendation ranking may all use probabilities, but the denominator that matters is different in each decision.
The metric table names each denominator so the operational risk is visible.
Pick the metric from the risk
- False alarm costWhen predicted positives consume scarce review, precision matters.
- Miss costWhen missed positives are dangerous, recall matters.
- Probability useWhen scores feed decisions or prices, calibration matters.
| metric | probability reading | risk emphasis |
|---|---|---|
| accuracy | P(correct) | overall label quality |
| precision | P(actual positive | predicted positive) | false alarms |
| recall | P(predicted positive | actual positive) | misses |
| calibration | observed frequency near stated score | probability trust |
Choose a metric for a review queue
A fraud team can manually review only 40 flagged transactions. The model flags 40, of which 30 are truly fraud; total fraud cases are 50.
- Queue precision30/40=0.75
- Fraud recall30/50=0.60
- Missed fraud50-30=20
- Risk readingprecision grades reviewer workload; recall grades missed fraud
Pro tip. The same model output can look acceptable or weak depending on whether the operational risk is false alarms or misses.
A fraud review team can inspect only 25 flagged cases. Which metric most directly grades how many inspected flags are real?
- Precision
- Recall
- Calibration
- Training loss
Precision conditions on predicted positives, exactly the limited review queue.
Notes
- Metrics summarize random outcomes over many predictions, so they should be read as frequencies or conditional probabilities.
- A calibrated score tries to match predicted confidence to observed frequency.
- Accuracy can be high or low without saying whether probability scores are honest.
Formulas
- Precision = TP / (TP + FP).
- Recall = TP / (TP + FN).
- Accuracy = (TP + TN) / total.
- Calibration compares predicted probability with observed frequency for similar scored cases.
Exam traps & shortcuts
- Precision and recall answer different conditional questions, so never swap their denominators mentally.
- Accuracy grades hard labels; calibration grades probability honesty.
Recap
Evaluation metrics are probability questions with business consequences.
- Denominator
- Precision, recall, and accuracy differ because they condition on different event worlds.
- Accuracy limit
- Accuracy ignores the probability confidence behind a hard label.
- Calibration
- A probability score is calibrated when stated confidence matches observed frequency.
- Risk
- Pick the metric whose denominator matches the cost of the decision.
Practise Probability Inside Evaluation Metrics
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 5-question practice set that ends the chapter
- 4 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device