Regression: Predicting Numbers
A line drawn through your examples
In this lesson
Regression is the first of the two supervised machine learning tasks, and it predicts a number. How much will this house sell for, how long will the delivery take, what score will this student get.
The idea is one you have seen on paper: draw a line through the dots and read off the answer. What follows is that, done arithmetically, and then the far more important question of how far off the line usually is.
Explain it like I’m 5
Regression is guessing a number by learning the pattern from old examples, drawing the line that comes closest to all your dots, then reading the answer off it.
The line of best fit
Given hours revised and the scores that followed, the pattern is a rising line. The line of best fit is the one passing as close as possible to every point at once.
NumPy will find it for you. np.polyfit(x, y, 1) fits a straight line (the 1 means “degree 1”) and returns the slope and the intercept. Those two numbers are the model.
import numpy as np
hours = np.array([1, 2, 3, 4, 5, 6])
scores = np.array([52, 55, 61, 64, 70, 72])
slope, intercept = np.polyfit(hours, scores, 1)
print("slope: ", round(float(slope), 2))
print("intercept:", round(float(intercept), 2))
predicted = slope * 7 + intercept
print("predict 7 hours:", round(float(predicted), 1))
errors = np.abs(np.polyval([slope, intercept], hours) - scores)
print("mean absolute error:", round(float(errors.mean()), 2))
slope: 4.23 intercept: 47.53 predict 7 hours: 77.1 mean absolute error: 0.78
Error is the honest part
Every prediction is wrong by some amount. The useful question is not “is it right?” but “how wrong is it, typically?”
Mean absolute error is the friendliest measure, and it works the same way whatever features you fed in: take each prediction, see how far it missed, ignore the direction, and average. An error of 0.78 marks on scores in the 50s and 70s is small. The same 0.78 predicting the number of children in a household would be useless.
That is the point: an error only means something next to the scale of the thing you are predicting. Always report it in the same breath as the prediction.
Use the fitted line to predict a score for 8 hours of revision, then compare it against what actually happened. The student scored 85. Work out the prediction from slope and intercept, and how far off it was.
import numpy as np
hours = np.array([1, 2, 3, 4, 5, 6])
scores = np.array([52, 55, 61, 64, 70, 72])
slope, intercept = np.polyfit(hours, scores, 1)
# TODO: predict the score for 8 hours of revision
prediction = 0
actual = 85
# TODO: how far off was the prediction? (always a positive number)
error = 0
print("predicted:", round(float(prediction), 1))
print("actual: ", actual)
print("off by: ", round(float(error), 1))The model is slope * hours + intercept, so the prediction is slope * 8 + intercept. The error is abs(prediction - actual), using abs because being 3.6 under is just as wrong as being 3.6 over.
import numpy as np
hours = np.array([1, 2, 3, 4, 5, 6])
scores = np.array([52, 55, 61, 64, 70, 72])
slope, intercept = np.polyfit(hours, scores, 1)
prediction = slope * 8 + intercept
actual = 85
error = abs(prediction - actual)
print("predicted:", round(float(prediction), 1))
print("actual: ", actual)
print("off by: ", round(float(error), 1))Where the line stops being trustworthy
Notice what just happened. The training data covered 1 to 6 hours, and the prediction for 8 hours was off by 3.6 marks, more than four times the typical training error of 0.78.
That is extrapolation, and it is the main way regression misleads. The line has no idea that scores cannot exceed 100, that revision has diminishing returns, or that nobody in the data revised for 8 hours. It just keeps going straight, forever, with total confidence.
Predict inside the range your examples covered and a simple model is often good. Predict outside it and you are trusting a straight line to describe a world that was never straight.
Common mistake: Assuming a straight line fits every problem
polyfit returns a line for any data at all, without complaint.
Plot the points first. If they curve, a straight line cannot describe them however much data you add.
Common mistake: Trusting predictions far outside the training range
The formula happily accepts any input.
Note the range your examples covered and treat anything beyond it as a guess, not a prediction.
Common mistake: Reporting a prediction with no error
The single number looks like the answer.
Always show the typical error next to it. Without it, nobody can tell whether to act on the number.
Common mistake: Measuring error only on the training data
It is the data you have, and the score comes out flattering.
That measures memory, not prediction. The next lesson but one is entirely about this.
What kind of answer does regression produce?
Prices, temperatures, durations. Predicting a category is classification instead.
What does mean absolute error measure?
Each miss is measured, the direction ignored, and the result averaged, so it is in the same units as the thing you are predicting.
Why are predictions outside the training range risky?
Extrapolation assumes the pattern holds where you have no evidence that it does.
Mini exercise (medium)
Fit a line to four house sales, sizes in square meters against prices in thousands, then predict the price of a 90 m² house and report the model's mean absolute error on the data it learned from. Finally, print whether 90 m² sits inside the range of sizes you trained on, since that decides how much to trust the answer.
Take the wheel. Complete the code, hit Run, and check your output right here.
import numpy as np
sizes = np.array([50, 75, 110, 135])
prices = np.array([150, 205, 295, 355])
# TODO: fit a straight line through the four sales
slope, intercept = 0, 0
# TODO: predict the price of a 90 m2 house
prediction = 0
# TODO: the mean absolute error on the training data
errors = np.array([0])
# TODO: is 90 inside the range of sizes we trained on?
inside = False
print("predict 90 m2:", round(float(prediction), 1))
print("mean absolute error:", round(float(errors.mean()), 2))
print("90 is inside the training range:", inside)np.polyfit(sizes, prices, 1) gives slope and intercept. Predict with slope * 90 + intercept. For the error, use np.polyval([slope, intercept], sizes) against the real prices, take np.abs, then .mean(). The range check is sizes.min() <= 90 <= sizes.max().
import numpy as np
sizes = np.array([50, 75, 110, 135])
prices = np.array([150, 205, 295, 355])
slope, intercept = np.polyfit(sizes, prices, 1)
prediction = slope * 90 + intercept
errors = np.abs(np.polyval([slope, intercept], sizes) - prices)
inside = bool(sizes.min() <= 90 <= sizes.max())
print("predict 90 m2:", round(float(prediction), 1))
print("mean absolute error:", round(float(errors.mean()), 2))
print("90 is inside the training range:", inside)