Bias and Fairness in Models

1Where bias enters a model

A supervised model learns the mapping its labels describe, and labels are records of past decisions, not of ground truth. If loan officers historically declined one community more often, 'was approved' encodes that behaviour, and a model fitted to it learns the behaviour with the same confidence as any real signal.

Bias also enters before labelling. A sampling frame that under-represents a group gives the model less to learn from exactly where reliability matters, and a measured feature can mean different things across groups — an arrest record measures policing intensity as much as offending.

None of this requires bad intent anywhere in the pipeline, which is why the response is measurement rather than reassurance: uneven treatment is found by computing rates per group, never by inspecting the code.

Figure. The highlighted node is the misread: training labels sit in the middle of the chain, treated by the fit as truth, while being themselves decisions somebody made. Whatever pattern the deciders had, the model inherits and repeats.

Where to look

1. Label provenanceAsk who produced the label and what it records. 'Approved' is a decision; 'repaid' is an outcome — models trained on each learn different things.
2. Sampling frameAsk who is missing or rare in the training rows; a group the data barely sees gets the least reliable predictions.
3. MeasurementAsk whether a feature means the same thing for every group, or measures the process that recorded it.

2Slice metrics: one number per group

A global metric is an average over everyone the model touches, and an average has no opinion about who is carrying the errors. Serious evaluation therefore slices: compute the same metric — accuracy, false positive rate, recall — separately for every group the decision lands on, and read the spread, not the mean.

Slices earn their own caveat: a rate on a small slice is noisy. Report the count next to every slice rate, and treat a startling number on thirty rows as a reason to collect data, not yet as a finding.

Figure. The dashed rule is the headline number, and it sits close to group A because group A is three times bigger. The overall metric is a weighted average whose weight is group size, not importance — which is exactly how a 61% slice hides inside an 85% headline.

Coding lab. Slice a metric by group runs in the app, with checks on your output.

3Selection rates and the four-fifths screen

For a yes/no decision the coarsest fairness number is the selection rate: of everyone in a group who was scored, what fraction got the favourable outcome. Disparate impact means the rates differ across groups badly enough to need an explanation, whatever the model intended.

The common screening heuristic — borrowed from US employment practice and useful well beyond it — is the four-fifths rule: flag the system when one group's selection rate falls below 80% of the highest group's rate. It is a tripwire, not a verdict: a flagged ratio starts an investigation into features and labels, and an unflagged one does not certify fairness.

Figure. The dashed line is four-fifths of the higher rate (0.8 × 60% = 48%). Group A's 30% sits well below it, so the screen trips. The bars are rates, not counts — 60 of 200 against 90 of 150.

Coding lab. Four-fifths screen in six lines runs in the app, with checks on your output.

4The two errors cost different things

Every binary decision system has two failure modes: the false positive (flagged, and should not have been) and the false negative (missed, and should not have been). They land on different people at different prices — a wrongly declined loan and a wrongly approved one are not the same event to anyone involved.

The threshold that trades them off is therefore a policy choice wearing a numeric costume. Choosing it means naming who bears each error, at what cost, and at what rate per group — which is why slice audits are run on false positive and false negative rates, not only on accuracy.

5Fairness metrics disagree by design

Once slices exist, several reasonable-sounding requirements compete. Demographic parity asks for equal selection rates across groups; equalised odds asks for equal false positive and false negative rates; calibration asks that a score of 0.8 mean the same 80% chance in every group.

These are not variants of one idea. A known impossibility result says that when the groups differ in base rate, a classifier cannot be calibrated and have equal error rates at the same time except in degenerate cases. The requirements are mutually exclusive, mathematically, not politically.

So 'is the model fair' is underspecified until you say fair by which criterion, and the criterion is chosen from the domain: what the decision does to people, which error is worse, and whether the score will be read as a probability.