Naive Bayes

1Classification as inference

A classifier answers 'which class produced this input?' — and Bayes rule makes that literal. Score each class by prior × likelihood, P(c) × P(x|c), and pick the class with the largest product; the normaliser P(x) is shared by all classes and drops out of the comparison.

A model built this way is called generative: it describes how each class generates inputs, then inverts that story with Bayes rule. The other family — discriminative models such as logistic regression, coming in the ML course — skips the story and fits the boundary P(c|x) directly.

The generative route pays off when the class-conditional structure is easy to estimate. Count how often each word appears in spam and in ham and you have the whole model: no gradients, no iterations — counting is the training loop.

Figure. The generative story naive Bayes commits to: first the class is drawn, then each word is generated from the class independently of the other words. Every arrow points from class to feature — the classifier runs the arrows backwards with Bayes rule.

2The independence assumption

P(x|c) for a whole document is unlearnable head-on — no corpus contains every sentence. Naive Bayes replaces it with the one assumption that makes counting sufficient: given the class, features are independent, so the document's likelihood is just a product of one small factor per word.

The assumption is false in almost every real dataset — 'free' and 'offer' plainly travel together in spam. What failure looks like is double counting: correlated words re-submit the same evidence, and the multiplied score comes out far too confident.

Yet the decision often survives, because classification uses only the argmax. Distorted probabilities that still rank the true class first cost nothing at decision time — naive Bayes is routinely a well-ranking, badly-calibrated model, a distinction the calibration lesson makes precise.

3Bag-of-words text

To use any of this on text you need features, and the crudest featurisation is the one naive Bayes thrives on: forget word order entirely and keep only which words occur, or how many times. A document becomes a long, sparse count vector over the vocabulary — the bag of words.

The fit between representation and model is exact. One likelihood factor per vocabulary word is precisely what the independence assumption knows how to combine, and sparse counts are exactly what add-one smoothing repairs.

Two standard variants: Bernoulli naive Bayes looks only at presence or absence of each word (and multiplies in absence factors for words that do not appear); multinomial naive Bayes uses the counts, which usually wins on longer documents. Word order, negation and phrasing are invisible to both — 'not good' and 'good' carry the same bag.

4Learning is counting (plus one)

Where do the priors and likelihoods come from? From labelled examples: the prior is each class's share of the training set, and each word's likelihood is its frequency within that class. This is why naive Bayes is supervised learning rather than applied probability — a different labelled sample yields a different classifier.

Raw frequencies have a fatal edge case. A word never seen in a class has probability zero there, and one zero vetoes the entire product no matter how strong the other evidence is — a single unseen word would make a mail certainly-not-spam.

Add-one (Laplace) smoothing pretends every word was seen once more than it was, so no factor is ever exactly zero. In practice implementations also work in log space — sums of logs instead of products — because a few hundred small factors multiplied together underflow floating point.

Figure. In the four spam mails, lottery never appears. Unsmoothed P(lottery|spam) = 0/4 zeroes the whole spam product. Add-one writes (0+1)/(4+2) = 1/6, so strong words elsewhere can still win.

From corpus to classifier

1. Count classesPrior = each class's share of the training set.
2. Count words per classLikelihood = the word's frequency within the class, plus one.
3. Sum logs at test timeScore each class by log prior plus log likelihoods; the argmax wins.

5Naive Bayes in code

The scikit-learn interface is three lines: construct, fit, score. GaussianNB is the variant for real-valued feature columns; for text you would reach for MultinomialNB or BernoulliNB on a count matrix.

The lab below stages the failure mode from the independence concept where you can watch it. Fit a Bernoulli model on eight tiny mails, then feed it the same evidence twice by duplicating a column: the reported probability climbs from 0.889 to about 0.97 — more confident on no new information — while the predicted class never moves.

That pair of observations — scores distorted, ranking intact — is the signature of correlated evidence in any probabilistic model, not just this one. It is worth internalising before the calibration lesson.

Figure. Construct, fit, score: GaussianNB for real-valued columns; MultinomialNB or BernoulliNB for counts. Duplicating a column climbs the reported probability from 0.889 to about 0.97 — more confident on no new information — while the predicted class never moves.

The interface

1. ConstructPick the variant for the feature type: Gaussian for real values, multinomial or Bernoulli for counts.
2. Fitclf.fit(X_train, y_train) does the counting.
3. Scoreclf.score(X_test, y_test) reports accuracy on rows the fit never saw.

Gaussian naive Bayes

from sklearn.naive_bayes import GaussianNB
clf = GaussianNB()
clf.fit(X_train, y_train)
print(clf.score(X_test, y_test))

Coding lab. Double-count the evidence runs in the app, with checks on your output.

6Where it wins, where it loses

Reach for naive Bayes when data is small, features are many, and an answer is needed now: it trains in one pass, has almost no hyperparameters, and is hard to beat on short-text classification. As a baseline it earns a place in any pipeline — a fancier model that cannot beat counting is telling you something.

It loses when correlated features double-count evidence badly enough to flip the ranking; when the numbers must be read as probabilities — its scores saturate toward 0 and 1; and when word order or feature interactions carry the signal.

The honest summary: trust its argmax sooner than its probabilities. What it would take to trust the probabilities is exactly the next lesson.