Linearity Assumptions

1What linearity assumes

Picture a model that predicts a flat's monthly rent from two facts about it: how many bedrooms it has, and which floor it sits on. A linear model makes that prediction the simplest way imaginable — it starts from a base amount and then adds a fixed amount for each unit of each feature. Something like: rent = ₹20,000 base, plus ₹3,000 for every bedroom, plus ₹500 for every floor you go up. Those fixed per-unit amounts (the ₹3,000 and the ₹500) are called the model's coefficients, or slopes — one number per feature, and that number is the whole of what the model believes about that feature.

Hidden inside that recipe is a strong promise called additivity: each feature adds its own fixed amount regardless of what the other features are doing. Under this promise, an extra bedroom is worth exactly ₹3,000 whether the flat is on the ground floor or the twentieth. The features never talk to each other — the model just sums their separate contributions. That is what "linear" is really assuming, and it is a claim about the world that can simply be false.

Real rents break the promise all the time. Suppose an extra bedroom is worth little on the ground floor but a lot on a high floor, because spacious high flats are penthouses with a premium. Now the effect of bedrooms depends on floor — the two features combine, and that combined effect is called an interaction. Here is the part beginners miss: a plain linear model can never discover an interaction on its own, no matter how much data you give it. It only has one slope per column you handed it, so if no column represents "bedrooms and floor together", the penthouse premium has nowhere to live. (The next concept in this topic shows the fix: you create that column yourself.)

Figure. The whole model is one sum: a ₹20,000 base, plus ₹3,000 for every bedroom, plus ₹500 for every floor, each flowing in on its own arrow. That per-unit amount is the feature's coefficient, and additivity is the missing arrow: nothing connects bedrooms to floor, so each contributes its fixed amount no matter what the other is doing.

2Traps, and the residuals that catch them

A second way the story goes wrong is outliers — individual rows whose values are wildly off the trend, often from a typo or a data-entry slip. The standard way of fitting the line, called least squares, picks the line that minimises the sum of squared misses: for each row it measures how far the prediction landed from the truth, squares that miss, and adds them all up. Squaring is the trap. A miss of 2 costs 4, but a miss of 20 costs 400 — a hundred times more. So one absurd row can dominate the total cost, and the fit will tilt the whole line toward that row just to shrink its enormous squared miss, distorting predictions for the many rows that were fine.

The third trap is collinearity: two feature columns that are near-duplicates of each other, like income in rupees and the same income in thousands. The data pins down their combined effect perfectly well, but it has no way to decide how to split the credit between the two — every split gives the same predictions. So the two coefficients can wander to enormous values with opposite signs, cancelling each other out. Predictions still look fine; it is the story "which feature mattered, and by how much" that has become meaningless. Whenever coefficients look absurd but accuracy looks normal, suspect near-duplicate columns.

How do you catch these failures in practice? Look at the residuals. A residual is just actual value minus predicted value, computed for each row — the amount the model missed by, positive when it underpredicted and negative when it overpredicted. The figure below plots each row's residual (vertical axis) against its fitted value, meaning the model's own prediction ŷ for that row (horizontal axis). If the linear story is honest, the misses are just random noise: a flat, patternless cloud hugging the zero line. A U-shaped bend means the straight line missed a curve in the data; a fan that widens to one side means the errors grow with the prediction. Any visible pattern is the residuals telling you the additive fixed-slope story is lying somewhere.

Figure. After a fit, plot residual against fitted value on a zero line through the middle. A healthy cloud is a flat random band around zero — no slope, no smile, no megaphone. A U or smile means the line missed a curve; a widening fan means variance grows with the prediction. Both say the additive constant-slope story is lying somewhere, even when predictions look mostly fine.

3First, a linear score

So far the model predicted a number that can be anything — a rent of ₹26,000, say. Now change the question: is this email spam, yes or no? There is no amount to predict, only two classes, and what you actually want is a probability — a number between 0 and 1 that says how confident the model is that the answer is yes. Logistic regression is the standard first tool for this, and the surprise is how little it changes: it keeps the entire linear machinery and bolts one small squashing function onto the end.

The linear part first, symbol by symbol. The model computes a score z = w \cdot x + b. Here x is the row's features — for the spam question, things like how many links the email contains and how many times it says "free". w is the weights, one number per feature, exactly like the rent slopes: each weight says how strongly its feature pushes the verdict toward spam (positive weight) or away from it (negative weight). b is the bias, a starting score before any feature is looked at. And z, the result of the weighted sum, is the raw score. It can be any real number — 3.7, or −12 — which is the problem, because a probability must live between 0 and 1.

Figure. Exactly the rent machinery, pointed at spam: each feature is multiplied by its own weight and summed with the bias b to give the raw score z. Positive weights push the verdict toward spam, negative ones away. The catch sits in the last node: z can be any real number — 3.7, or −12 — and a probability must live between 0 and 1.

4The squash and its landmarks

The fix is the sigmoid function: \sigma(z) = 1/(1+e^{-z}), where e \approx 2.718 is the mathematical constant. You do not need to memorise why this formula, only what it does: it takes any real score and squashes it into the range between 0 and 1, so the output can honestly be read as "the probability that this email is spam". Feed in a huge positive score and e^{-z} shrinks toward zero, so the output creeps toward 1. Feed in a huge negative score and e^{-z} blows up, so the output creeps toward 0. The output never quite touches 0 or 1 — the model never claims absolute certainty.

Three landmark values anchor your intuition. At z = 0 — the features push neither way — the sigmoid gives exactly 0.5: a coin flip. At z = 2 it gives about 0.88; at z = -2, about 0.12. Notice the curve is steep near zero and nearly flat far from it: moving the score from 0 to 2 changes the probability a lot, while moving from 8 to 10 barely changes it, because the curve has already saturated near 1. Extra evidence matters most when the model is still unsure.

Figure. Adapted from undergrad_ml ml_linear_models/logreg. The linear score z = w·x + b is still a weighted sum; the sigmoid only squashes it into (0, 1). At z = 0 the probability is exactly 0.5 — the usual decision cut — and far from zero the curve saturates so ever-larger |z| barely moves p. Order of scores is preserved because the map is monotonic.

5The boundary stays straight

Here is the point beginners trip on. The sigmoid's graph is a curved S-shape, so it feels like the model's dividing line between "spam" and "not spam" must be curved too. It is not. The usual rule is: predict spam when the probability exceeds 0.5, and the probability is exactly 0.5 precisely when z = 0. So the dividing line — called the decision boundary — is the set of feature values where w \cdot x + b = 0, and that is a linear equation: a straight line with two features, a flat plane with three. The S-curve lives in the graph of probability against score; the boundary lives in feature space, and it stays straight.

One last useful property: the sigmoid is monotonic, meaning a bigger score always produces a bigger probability, with no exceptions. So if one email scores z = 3 and another scores z = 1, the first is guaranteed to get the higher spam probability. When all you need is a ranking — say, sorting emails from most to least suspicious — the raw scores z already carry the full ordering, and the sigmoid changes none of it.

6Fixing broken additivity

Back to the rent model with the penthouse problem. The first concept showed the symptom: additivity — the promise that each feature adds a fixed amount on its own — is false, because an extra bedroom is worth more on a high floor than on the ground floor. You can see it in the residuals, the per-row misses (actual rent minus predicted rent): instead of scattering randomly, they cluster by combination — the model consistently underpredicts flats that are both large and high, and the misses line up with that specific pairing of features. That clustering is the diagnosis. Now the fix, and it is far less drastic than beginners expect.

The fix is: build the missing column yourself. Take each row, multiply its bedrooms value by its floor value, and store the result as a brand-new feature column — for a 3-bedroom flat on floor 10, the new column holds 30. This is called an interaction term (or product column). Then re-fit the same linear model with three feature columns instead of two: bedrooms, floor, and bedrooms × floor. The model gives the new column its own slope, just like any other column, and that slope is exactly where the penthouse premium can now live — the new column is large only when both bedrooms and floor are large, which is precisely the situation the old model could not express.

Figure. Additivity keeps a fixed bedroom slope at every floor: the dashed high-floor line stays parallel to ground. A bed-times-floor product is still one linear column, but it tilts the high-floor prediction so the lines fan — the penthouse premium the plain additive fit cannot invent.

7Still linear, and still overfittable

"But wait — isn't the model no longer linear if it contains a multiplication?" This is the subtlety worth slowing down for. "Linear" in linear model means linear in the parameters: the prediction is still a plain weighted sum of columns — (slope₁ × column 1) + (slope₂ × column 2) + (slope₃ × column 3) + base. The model neither knows nor cares that you built column 3 by multiplying two others; to the fitting procedure it is just one more column of numbers to assign one slope to. All the cheap, reliable machinery of linear fitting still applies. What changed is the vocabulary of columns you offered — and that part is your job, not the algorithm's. A linear model never invents a product feature; it happily fits one you author.

One warning before you go inventing columns freely. Every column you add is another dial the model can turn, and with enough dials a model can memorise the accidental quirks of your training data instead of learning the real pattern — that failure is called overfitting. The guard is to check the new column against data the model never trained on (a held-out validation set): if the interaction cuts error on the training rows but the validation error barely moves, the new column is memorising noise, not capturing a real premium. Techniques that deliberately restrain weights — regularisation — apply to interaction slopes like any others; they get a full treatment in the next topic. The steps strip below turns this into a routine: spot the residual cluster, author the column, verify it earns its keep on validation.

1. Step 1Plot residuals vs each feature and vs plausible products (bed×floor, income×debt).
2. Step 2Add the interaction or bucket column on train only; re-fit and compare validation error.
3. Step 3If the new term barely moves validation but train jumps, you overfit — try stronger penalty or more data.