Random Forests

1Bagging and forests

Imagine you are about to buy a used car, and you want an answer to one question: will this car break down within a year? You could ask a single mechanic who has memorised every car he has ever worked on. The trouble with that mechanic is that he memorised too well — he fixates on quirks. "The last silver 2014 hatchback I saw was a lemon, so this one must be too." That is exactly what a single decision tree does when you let it grow as deep as it likes: it keeps splitting until it has memorised the training data, real patterns and meaningless quirks alike, and the quirks come along for the ride on new cars.

A better plan is to assemble a committee — say, 100 mechanics — and let them vote. But here is the catch a beginner should pause on: a committee only helps if its members disagree in useful ways. If all 100 mechanics trained on the same records in the same way, they would all inherit the same quirks and repeat the same mistake, just 100 times louder. So the whole trick of this lesson is to deliberately train each committee member a little differently. Random forests do this in two ways, and each has a name worth learning.

Figure. One mechanic who memorised too well carries his quirks onto the next car, and a committee of 100 identically trained mechanics just repeats the same mistake 100 times louder. The committee only helps when each member is deliberately trained a little differently.

2Two ways to make trees disagree

The first source of disagreement is the bootstrap sample — the rows each tree gets to study. Suppose you have a pile of 1000 past car records. To build one mechanic's study pile, you draw one record at random, copy it into their pile, and then put it back before drawing again. "With replacement" means exactly that putting-back step. After 1000 draws, that mechanic's pile also holds 1000 records, but some records were drawn twice or three times, and roughly a third of the originals were never drawn at all. Do this separately for every mechanic and each one has studied a different remix of the same history. Training many models on bootstrap samples and then combining their answers is called bagging — short for "bootstrap aggregating".

The second source of disagreement is the random feature subset. A feature is just one column of the data — mileage, age, brand, engine size. Every time a tree is about to ask a splitting question ("is mileage above 80,000?"), a random forest hands it only a random handful of the columns to choose from: this split it may look at mileage and age, the next split perhaps brand and engine size. Why do that? Because if one column is very strong, every tree would otherwise open with the same question and grow into near-copies of each other — and near-copies are exactly the useless committee from before. Hiding some columns at each split forces the trees down genuinely different paths.

Figure. One mechanic's study pile comes from 1000 draws with replacement over the 1000 original records: some records land in the pile twice or three times, while roughly a third are never drawn at all — so each mechanic studies a different remix of the same history.

3The committee hands in its answer

Now the committee answers. Each tree hands in its own prediction for the car in front of you. The symbol \hat{y} — read "y-hat" — is the standard shorthand for a model's guess at the true answer y, so in the diagram tree 1's guess is \hat{y}_1, tree 2's is \hat{y}_2, tree 3's is \hat{y}_3. For a classification question (breaks down: yes or no) the forest takes a majority vote: if 70 of 100 trees say "breaks down", the forest says "breaks down". For a regression question (predict a number, like resale price) there is nothing to vote on, so the forest simply averages the 100 numbers.

Figure. Three shallow trees each cast one prediction; bagging's vote or average turns those disagreeing calls into a single forest ŷ, so one overgrown path cannot dominate.

1. Bootstrap rowsDraw each tree's training set with replacement — some rows repeat, some are left out.
2. Grow diverse treesRandom feature subsampling at each split keeps trees from copying the same path.
3. Vote or averageClassification uses majority vote; regression averages the tree outputs.

4What averaging buys, and what it costs

Why does averaging disagreeing trees help? The failure it attacks is called variance — how much a model's answers would wobble if you re-collected the training data and trained again. One overgrown tree has high variance: change a few rows and its memorised path changes wildly. But each tree's quirks came from its own remixed pile and its own hidden columns, so the quirks point in different directions — and when you vote or average, quirks pointing in different directions largely cancel, while the real pattern, which every tree saw, survives. One tree's weird path is diluted by hundreds of others.

There is a price, and it is honesty to name it: interpretability. With a single tree you could point at one readable path — "high mileage, then old battery, therefore lemon" — and show a customer exactly why. With 100 trees there is no single path to point at; the answer is a vote. Forests offer feature-importance scores as a consolation (a summary of which columns the trees split on most usefully), but treat those as a sketch that guides your attention, not a proof of what causes what.

5Lab: tree vs forest

Now watch the committee beat the lone expert with your own eyes. The lab fits two models on exactly the same data: a DecisionTreeClassifier with max_depth=None, which is the memorising mechanic — a tree allowed to grow without any depth limit — and a RandomForestClassifier with n_estimators=50. That last name deserves a plain-words gloss: n_estimators is simply the number of trees in the committee, so this forest is 50 trees, each grown on its own bootstrap sample with random feature subsets at each split.

Where does the data come from? The function make_classification manufactures a practice table on the spot: 600 rows, 8 feature columns, and a 0-or-1 label for each row. In the code, X is that table of input columns and y is the list of correct labels — the same X-and-y convention every sklearn model uses. Because the table is synthetic, the columns have no real-world meaning, and that is fine here: the point of this lab is to compare two models on identical ground, not to learn about cars.

Before either model is fitted, train_test_split sets aside 25% of the rows as a test fold — rows neither model is allowed to see while training. Why insist on that? Because scoring a model on rows it trained on is like grading a student on questions they studied with the answer key attached: the memorising tree would look perfect and you would learn nothing. Calling .score(X_test, y_test) afterwards returns the fraction of those never-seen rows each model predicts correctly, which is the number that actually matters.

The lab runs as two cells, and the mechanics are worth spelling out. Cell 1 builds the data, fits both models, and prints both scores. Every variable it creates — forest, X, tree_score, forest_score — stays alive in the shared kernel, which is the Python session running behind all the cells. That is why Cell 2 can draw a bar chart of tree_score against forest_score without refitting anything: it just reads the numbers still sitting in the kernel. Want a different title or different bar colours? Edit Cell 2 and re-run only that cell — Cell 1's work is untouched.

What should you expect to see? The forest's test score should match or beat the unlimited tree's, and — if you re-run Cell 1 with a different random_state — the forest's score should wobble less from run to run, because averaging 50 trees is exactly the variance cure the previous concept promised. Two ground rules while you experiment: keep n_samples in the hundreds so the browser runtime stays responsive, and never score either model on rows it was fitted on — the test fold exists precisely so you don't.

No diagram — the score comparison is drawn by the coding lab plot, not a static figure.

1. Fit a lone treeTrain DecisionTreeClassifier with no depth cap on the same split — note train vs test gap.
2. Fit the forestTrain RandomForestClassifier(n_estimators=50) on identical X_train, y_train.
3. Compare hold-outForest test score should beat or match the single tree with less variance run to run.

Coding lab. Compare a tree to a random forest runs in the app, with checks on your output.