Train/Test Split and Overfitting
Do not grade the exam with the answer key
In this lesson
The last two lessons both ended with a warning: a score measured on the training data does not mean what it looks like. This lesson fixes that, and it is the single most important idea in the unit.
The principle is one every student already knows. If you sit an exam having been given the answer key, your mark says nothing about whether you understood the subject.
Explain it like I’m 5
Training is studying; testing is taking a quiz with questions you have not already seen. A model that only saw the answers has memorized, not learned.
Hold some examples back
The fix is almost too simple. Before training, put some of your examples aside and do not let the model see them. Train on the rest. Then score the model on the held-back rows.
Those two groups are the training set and the test set. A common split is 80/20: most of the data to learn from, a fifth kept back to be honest with.
The score on the test set is the number worth reporting, because those rows are a stand-in for the future data the model will actually face.
Do the split by hand so the mechanics are clear. Take the first 80% of the examples as the training set and the rest as the test set, then confirm that nothing appears in both, because a single leaked row is enough to make the test score dishonest.
examples = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]
# TODO: split at 80% - the first part trains, the rest is held back
split_at = 0
train = []
test = []
print("train:", train)
print("test: ", test)
print(f"{len(train)} to train on, {len(test)} held back")
print("no overlap:", set(train).isdisjoint(test))split_at = int(len(examples) * 0.8) gives 8. Then slice: examples[:split_at] for training and examples[split_at:] for testing. set(train).isdisjoint(test) proves no example is in both.
examples = ["a", "b", "c", "d", "e", "f", "g", "h", "i", "j"]
split_at = int(len(examples) * 0.8)
train = examples[:split_at]
test = examples[split_at:]
print("train:", train)
print("test: ", test)
print(f"{len(train)} to train on, {len(test)} held back")
print("no overlap:", set(train).isdisjoint(test))Watching a model memorize
Now the demonstration. Two models on the same six training examples: a straight line, and a much more flexible curve that is allowed to bend four times.
The flexible curve can pass closer to every training point: it has more freedom. Watch what that flexibility costs on the two examples held back.
import numpy as np
hours_train = np.array([1, 2, 3, 4, 5, 6])
scores_train = np.array([52, 55, 61, 64, 70, 72])
hours_test = np.array([7, 8])
scores_test = np.array([79, 85])
straight = np.polyfit(hours_train, scores_train, 1)
wiggly = np.polyfit(hours_train, scores_train, 4)
def error(model, x, y):
return float(np.abs(np.polyval(model, x) - y).mean())
print("straight line train error:", round(error(straight, hours_train, scores_train), 2))
print("straight line test error: ", round(error(straight, hours_test, scores_test), 2))
print("wiggly curve train error:", round(error(wiggly, hours_train, scores_train), 2))
print("wiggly curve test error: ", round(error(wiggly, hours_test, scores_test), 2))
straight line train error: 0.78 straight line test error: 2.75 wiggly curve train error: 0.53 wiggly curve test error: 14.95
Overfitting, underfitting, and generalization
Three words describe the whole balance, and they are the vocabulary every machine learning discussion assumes.
Overfitting is the wiggly curve: the model learned the training examples too specifically, including their accidents, and cannot cope with anything new. The signs are a near-perfect training score and a poor test score.
Underfitting is the opposite, a model too simple to capture the real pattern. Both scores are bad. Fitting a straight line to something genuinely curved underfits, and adding data will not save it.
Generalization is the goal in between: doing well on data you have never seen. That, and not the training score, is what “a good model” means.
The usual cures for overfitting are more training data, or a simpler model. The usual cure for underfitting is a more flexible model, or better features.
Common mistake: Testing on the same data used for training
It is the data you have, and holding some back feels wasteful.
Split before training. A score on seen data measures memory, not prediction.
Common mistake: Celebrating a perfect training score
100% looks like success by every normal standard.
On training data, perfection usually means memorization. Check the test score before believing anything.
Common mistake: Adjusting the model until the test score improves
It feels like ordinary iteration, and the number does go up.
You are fitting to the test set. Use a separate validation split for choosing, and save the test set for a single final check.
Common mistake: Assuming a complex model is a better one
More flexibility sounds strictly more capable.
Flexibility is what allows memorization. On small datasets the simpler model usually generalizes better, as it did above.
What is overfitting?
The signature is a very low training error beside a much larger test error.
Why hold back a test set?
Held-back rows stand in for the future data the model will actually meet.
What does generalization mean?
It is the actual goal. The training score is only a means of getting there.
Mini exercise (hard)
Compare three models of increasing flexibility (degree 1, 2 and 5) on the same six training points and two held-back points. Print each model's training and test error, then say which degree generalizes best. The answer is not the one with the lowest training error.
Your turn. Fill in the code below and press Run to test it right here, nothing to install.
import numpy as np
hours_train = np.array([1, 2, 3, 4, 5, 6])
scores_train = np.array([52, 55, 61, 64, 70, 72])
hours_test = np.array([7, 8])
scores_test = np.array([79, 85])
def error(model, x, y):
return float(np.abs(np.polyval(model, x) - y).mean())
results = {}
# TODO: for each degree, fit the model and print its train and test error,
# recording the TEST error in `results`
for degree in (1, 2, 5):
pass
# TODO: pick the degree with the smallest test error
best = 0
print("best on test data: degree", best)Loop over [1, 2, 5], fit each with np.polyfit(hours_train, scores_train, degree), and reuse an error() helper for both sets. To pick the winner, choose the degree with the smallest test error using min(...) with a key.
import numpy as np
hours_train = np.array([1, 2, 3, 4, 5, 6])
scores_train = np.array([52, 55, 61, 64, 70, 72])
hours_test = np.array([7, 8])
scores_test = np.array([79, 85])
def error(model, x, y):
return float(np.abs(np.polyval(model, x) - y).mean())
results = {}
for degree in (1, 2, 5):
model = np.polyfit(hours_train, scores_train, degree)
train_err = error(model, hours_train, scores_train)
test_err = error(model, hours_test, scores_test)
results[degree] = test_err
print(f"degree {degree}: train {train_err:6.2f} | test {test_err:8.2f}")
best = min(results, key=results.get)
print("best on test data: degree", best)