Machine Learning · Machine Learning Core
Train / Test Split
Hold out train, validation, and test folds so fit, tune, and final score never share rows — then cut them with a seeded sklearn split in the browser.
Before any model sees labels, cut the table into roles that never share rows. Train fits parameters, validation tunes choices, and test reports final generalisation — opened once, at the end. This lesson locks that contract, then cuts a seeded hold-out in the browser.
- Machine Learning
- Easy level
- 5 concepts
- 11 practice questions
1Train / validation / test
Imagine a teacher who hands the class a booklet of 100 practice questions, and then sets the final exam using those exact same 100 questions. Every student scores near 100% — but the scores are meaningless, because the students may have simply memorised the answers rather than learned the subject. To find out who actually learned, the exam must use questions the students have never seen. Machine learning has exactly the same problem: a model that is scored on the same rows it studied looks brilliant while proving nothing. The fix is called a train / test split, and it is the first thing you do with any dataset — before any model exists.
Make it concrete. Suppose you have a spreadsheet of 1,000 rows, where each row is one house that already sold: its size, its number of bedrooms, and the price it sold for. You want a model that predicts price for a new house. Instead of showing the model all 1,000 rows, you cut the spreadsheet — once, up front — into three piles with three different jobs. The training set (say 700 rows) is the pile the model studies: it is the only data used to fit the model's internal numbers, called its parameters. Make 'parameters' physical: for a straight-line model predicting price from size, the slope of that line is a parameter — you never type it in, the fitting process writes it by itself, from the training rows. But a model also has settings the fitting process cannot write, because they must already be fixed before fitting starts — the way an oven's temperature has to be set before the cake goes in, not by the cake while it bakes. How deep may a decision tree grow? How strongly should complexity be penalised? The fit cannot choose those; someone has to pick them by hand, up front. Settings of that kind are called hyperparameters, to separate them from the parameters the fit writes on its own.
The validation set (say 150 rows) is a pile the model never studies, and it exists to referee exactly that hand-made choice. Here is one complete tuning walk, because 'tuning' stays foggy until you watch it happen. Unsure how deep to let the price-predicting tree grow, you train three separate trees on the same 700 training rows: one capped at depth 2, one at depth 5, one at depth 10. Each tree then predicts prices for the 150 validation rows it has never seen, and you write down how far off it typically was: the depth-2 tree misses by about ₹4.1 lakh (too shallow to catch the real patterns), the depth-5 tree by ₹2.9 lakh, and the depth-10 tree by ₹3.6 lakh (deep enough to memorise training quirks that do not repeat). Depth 5 wins. That loop — try a few settings, score each on the validation pile, keep the winner — is everything the word 'tuning' means. And notice what the test set (the last 150 rows) did during the whole contest: nothing. It stays locked in a drawer until the contest is over, then it is opened exactly once, to score only the winning depth-5 tree — and that one score is your honest answer to 'how well will this model do on data it has never seen?' That ability to perform on unseen data is called generalisation. No row is ever allowed to sit in two piles at once.
Figure. One dataset, cut once: the big share fits parameters (train), a slice tunes choices like depth or learning rate (validation), and the last slice is opened once, at the end, to report generalisation (test). Shuffle before cutting - unless order carries meaning, as in time series, where the cut must respect time.
| Role | Job | Open when |
|---|---|---|
| Train | Fits parameters | During every fit |
| Validation | Tunes choices (depth, regularisation, threshold) | While choosing among settings |
| Test | Reports final generalisation | Once, at the end — never for choice |
One tuning walk: three depths, one winner
You are tuning one hyperparameter — how deep the price-predicting tree may grow — with candidate depths 2, 5 and 10, using the 700-row training pile, the 150-row validation pile and the 150-row test pile cut from the 1,000-row spreadsheet of sold houses. Walk the contest through to the one number you are allowed to report.
- Train three trees on the same 700 training rows, one per candidate depththree fitted trees: depth 2, depth 5, depth 10
- Each tree predicts prices for the 150 validation rows it never studied; record the typical missdepth 2: ₹4.1 lakh · depth 5: ₹2.9 lakh · depth 10: ₹3.6 lakh
- Keep the setting with the smallest validation missdepth 5 wins — 2 was too shallow, 10 memorised training quirks
- Only now open the test drawer and score the one winner on its 150 rows, oncethe depth-5 tree's test score — the only number you report
Pro tip. The three validation scores exist to make one choice; none of them is the score you quote. The test pile met only the finished winner — score all three depths on test instead, and the test pile stops being held out.
Why must final model performance never be reported on data used during training or hyperparameter selection?
- Test sets automatically rebalance skewed class proportions across features
- Training scores measure memorisation of seen rows, whereas only an untouched test split estimates true generalisation
- Scikit-learn throws an unhandled exception if evaluated on training arrays
- Training loss is always mathematically larger than held-out test error
A model can easily overfit and score 100% on data it was trained on. Only an isolated, untouched test set provides an unbiased estimate of real-world performance.
2When a pile stops being held out
Here is the rule that trips up almost everyone, so read it twice: a pile only counts as 'held out' while no decision depends on it. Suppose you try forty different hyperparameter settings, score each one on the test set, and keep the best. It feels harmless — but by picking the winner using test scores, you have quietly tuned your model to that particular test set, and its score is now optimistic: flattering on these 150 rows, unreliable on genuinely new ones. That comparing-and-picking job is precisely what the validation set exists for. The test set answers one question, once, and never influences a choice.
Figure. Comparing forty settings and keeping the best is validation's job. The test pile meets only the finished winner, exactly once — score the candidates on test and you have quietly tuned to it, and its number turns optimistic.
You try 12 tree depths, pick the depth with the best test score, and quote that score as the model's generalisation. What went wrong?
- Nothing — the test fold existed exactly for comparing settings
- The test set stopped being held out the moment a choice depended on it; depth belonged on validation
- Twelve settings is too few for the comparison to be fair
- The depths should have been compared on the training score instead
A set is held out only while no decision depends on it. Choosing among settings is validation's job; after tuning on test, the quoted number is optimistic.
3Shuffle, time, and repeat customers
One more decision before you cut: should the rows be shuffled first? If your rows are interchangeable — 1,000 houses in no meaningful order — then yes, shuffle, so each pile gets a fair random mix rather than, say, all the houses from one neighbourhood. But if the order of rows carries meaning — daily sales with timestamps, seasons, one customer's transactions in sequence — a random shuffle cheats: the model would train on December rows and be tested on January rows, effectively peeking at the future. For time-ordered data, cut by time instead, so the test pile is the future the model will actually face in real use.
Sneaky overlap can also cross the cut even when no single row is duplicated. Say your job is to predict each customer's next month, and customer 4127 has some rows in the training pile and some in the test pile. The model meets that customer's personal habits during training, then gets 'tested' on the same customer — so the test rows are not really unseen. When rows belong to a person, a store, or any repeating entity, keep each entity entirely on one side of the cut.
| Your rows | How to cut | Why |
|---|---|---|
| 1,000 houses in no meaningful order | Shuffle, then cut | Each pile gets a fair mix, not one neighbourhood |
| Daily sales with timestamps | Cut forward in time | Test must be the future — a shuffle trains on December and tests on January |
| Customer 4127's rows spread across months | Keep each customer wholly on one side | The fit must not meet the tested customer's own habits |
Cutting whole customers: when 150 is impossible
Your job is predicting each customer's next month, so no customer may straddle the cut — every customer's rows must land wholly in one pile, or the model meets its examinees before the exam. The table holds 1,000 rows from 40 customers: 10 heavy customers with 40 rows each and 30 light customers with 20 rows each. You want the lesson's 700 / 150 / 150 arrangement. Work out whether a leak-free 150-row test pile can exist at all, and if not, build the nearest arrangement that keeps every customer whole.
- Check the table first: 10 heavy customers × 40 rows plus 30 light customers × 20 rows is 400 + 6001,000 rows
- Any pile built from whole customers holds some number of heavies (call it a) and some lights (call it b), so its row count is 40a + 20b = 20 × (2a + b) — always a multiple of 20. Test the target: 150 ÷ 207.5 — not whole, so no whole-customer pile can hold exactly 150 rows
- Flip that around: every possible 150-row test pile therefore splits at least one customer, leaving that customer's remaining rows in train — the fit absorbs the tested customer's own habits. No shuffle, however lucky, escapes this; it is arithmetic, not chancea leak-free 150-row test pile does not exist
- Move to the nearest multiples of 20: 140 or 160 rows. Take 160 for test, built as 1 heavy customer + 6 light customers: 40 + 6 × 20 = 40 + 120160 rows
- Validation the same way, from different customers: 1 heavy + 6 light, 40 + 120160 rows
- Train takes everyone left: 10 − 2 = 8 heavy customers and 30 − 12 = 18 light customers, so 8 × 40 + 18 × 20 = 320 + 360680 rows
- Check nothing vanished or doubled: 680 + 160 + 160 rows, and 8 + 1 + 1 heavy customers, 18 + 6 + 6 light1,000 rows, 10 heavy, 30 light — every customer whole, in exactly one pile
- Report the shares honestly: 680 ÷ 1,000, then 160 ÷ 1,000 twice68% / 16% / 16% — near, not equal to, 70/15/15
Pro tip. The drift from 70/15/15 to 68/16/16 is not sloppiness — it is the price of the group rule, and it is the honest buy. A test pile that hits 15.0% exactly by splitting one customer across the cut is scoring a model that has already met its examinees; a 16% pile of wholly unseen customers measures what production will actually face.
You are building a stock price forecaster over 5 years of daily prices. Why should you avoid a uniform random shuffle before splitting?
- Random shuffling changes the variance of the target price column
- Time-series datasets must contain strictly fewer than 100 observations
- Scikit-learn requires all time features to be formatted as Unix epochs
- Random shuffling leaks future market movements into the training set, violating temporal causality
For temporal data, training data must strictly precede testing data chronologically. Shuffling allows past predictions to be made using future data, creating severe leakage.
4One function makes the cut — pin its seed
In these labs the cutting tool is a ready-made function from the scikit-learn library (imported in Python as sklearn): it is called train_test_split, and it shuffles your rows and cuts them in one call. It takes an option named random_state — a number that fixes the shuffle, so the 'random' cut comes out identical every time. Set it (any fixed number, like 0 or 42, works) and two classmates — or you, tomorrow — get exactly the same piles and can compare scores fairly. Leave it unset and every run cuts a different test pile; on a small pile of 150 rows, just a few rows landing differently can swing the score by several points, making runs impossible to compare.
The function also takes test_size, the fraction of rows to hold out: test_size=0.25 means one quarter of the rows become the test pile and the remaining three quarters become training. Notice that gives you only two piles. To get all three, cut twice: first split off a temporary pile (train versus 'the rest'), then split that temporary pile into validation and test — pinning random_state on both cuts, so the whole arrangement reproduces exactly.
Figure. train_test_split returns two piles per call, so cut twice for three: first split off a temporary pile (train versus 'the rest'), then split that pile into validation and test — pinning random_state on both cuts so the whole arrangement reproduces exactly.
- Shuffle or respect timeExchangeable rows get a random shuffle; timestamped rows get a forward cut so test is the future.
- Cut once into rolesTrain fits, validation tunes, test reports — no row appears in two folds.
- Pin the seedSet random_state on train_test_split so the same hold-out reproduces on every re-run.
| Option | What it does | The slip if you get it wrong |
|---|---|---|
| random_state | Pins the shuffle so every re-run cuts identical piles | Unseeded, a 150-row test pile re-draws each run and the score swings by points |
| test_size | Fraction held out — of the rows that call receives | 0.15 on the 300-row second cut holds out 45 rows, not 150 |
Two cuts to reach 700 / 150 / 150
You want the lesson's headline arrangement — 700 training rows, 150 validation rows, 150 test rows out of the 1,000-row spreadsheet of sold houses — but train_test_split makes only two piles per call, and its test_size option is a fraction. What fraction do you pass on each of the two calls, and why is the second one not 0.15?
- Target sizes first: 0.70 \times 1000 = 700 train, 0.15 \times 1000 = 150 validation, 0.15 \times 1000 = 150 test700 / 150 / 150
- Cut 1 splits train from everything else. 'Everything else' is validation plus test together: 150 + 150 = 300 rows, so test_size = 300 / 10000.30 → 700 train, 300 'the rest'
- Cut 2 splits only the 300-row rest, so its test_size is a fraction of those 300 rows, not of the original 1,000: 150 / 3000.50 → 150 validation, 150 test
- Check nothing was lost or doubled: 700 + 150 + 1501,000 — every row in exactly one pile
Pro tip. Passing test_size=0.15 on the second call is the classic slip: 0.15 of the 300-row remainder is 0.15 \times 300 = 45 rows, so you would get 255 validation and 45 test instead of 150 each. Each call's fraction is measured against the rows that call receives. And pin random_state on both calls — seed only the first and cut 2 shuffles a different remainder every run, quietly changing which rows sit in your test pile.
What is the primary engineering reason to pass a fixed integer to random_state in train_test_split?
- It speeds up array computation by pre-allocating contiguous memory buffers
- It guarantees that the test set achieves the highest possible accuracy score
- It normalises all feature columns to have zero mean and unit variance
- It makes the pseudorandom partition deterministic, ensuring identical train and test splits across runs
Setting random_state ensures reproducible splits so that variations in test scores reflect actual model changes rather than lucky or unlucky data slices.
5Lab: hold out a test fold
Time to make the cut yourself. This lab builds a small practice table of made-up data, splits it with train_test_split, and checks the result — no model is trained anywhere in this notebook, because the lesson is the cut itself, not a score.
First, meet the variable names, because scikit-learn code uses them everywhere and never explains them. X (capital X) is the table of inputs: one row per example, one column per measured feature — here 400 rows and 6 columns of numbers. y (small y) is the answer column: one label per row, a 0 or a 1 saying which class that row belongs to. Calling train_test_split(X, y, ...) cuts both together, keeping each row paired with its own label, and hands back four pieces: X_train and y_train (the rows and labels the model would study) and X_test and y_test (the held-out rows and labels). The suffix tells you the pile; the letter tells you inputs versus answers.
Cell 1 does the work: make_classification manufactures the practice table (n_samples=400 rows, n_features=6 columns), train_test_split cuts it, and two print lines confirm the shapes — with test_size=0.25 you should see 300 rows in train and 100 in test, and .shape shows each as (rows, columns) so you can check the arithmetic yourself. Cell 2 draws a scatter plot of the training pile only; it deliberately does not rebuild the data, because both cells share one running Python session (the kernel), so X_train and y_train created in Cell 1 are still alive when Cell 2 runs. Run Cell 1 first, then Cell 2 — then change the markers or the title and re-run just Cell 2 to see the plot update without re-cutting anything.
Two habits to carry out of this lab. Pin random_state (both make_classification and train_test_split take it) so every re-run cuts the identical piles and your numbers are comparable across runs. And treat X_test and y_test as frozen the moment they exist: do not plot them, summarise them, or peek at them — in a real project they stay untouched until the one final score. Keep n_samples in the hundreds, too; this notebook runs in your browser, and small tables keep it snappy.
No diagram — the split is drawn by the coding lab plot, not a static figure.
| Step | Why |
|---|---|
| Call train_test_split with a seed | Same fold every re-run |
| Print train and test shapes | Confirm the cut before any fit |
| Leave X_test / y_test untouched | Final score must use rows the fit never saw |
Coding lab. Hold out a test fold runs in the app, with checks on your output.
A classmate runs the split lab twice without random_state and gets test scores of 0.86 and 0.79 from identical code. What explains the gap?
- The model's weights are randomly initialised
- Each run cut a different hold-out, and on a small fold a few moved rows shift the score by points
- sklearn retrains differently on each call
- The browser kernel cached stale variables between runs
An unseeded train_test_split draws a different test fold every run. Pinning random_state reproduces the same cut, so re-runs are comparable.
Notes
- Cut the table once into roles that never share rows: train fits, validation tunes, test reports generalisation once.
- A set stops being held out the moment a decision depends on it — pick hyperparameters on validation, not on test.
- Pin random_state on train_test_split so two runs (or two classmates) score the same hold-out.
Exam traps & shortcuts
- Keep lab datasets under 2000 rows in the browser runtime.
- Split train and test before fitting any model that sees labels.
Recap
This lesson in brief:
- Three roles, no shared rows
- Train fits parameters, validation tunes choices, and test reports final generalisation — opened once, at the end.
- Seed the cut
- Pin random_state on train_test_split so classmates (and you tomorrow) score the same hold-out.
Practise Train / Test Split
Reading is free and needs no account. Practice, mocks and progress live in the app.
- 11 exam-style questions on this topic, with explanations
- A 5-question practice set that ends the chapter
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device