Classification: Predicting Categories

Sorting new things using old examples

Advanced 13 min

In this lesson

The other half of supervised machine learning. Classification puts a new example into one of a fixed set of buckets: spam or not spam, which species, will this customer renew.

You are going to write a working classifier in about five lines. Doing it by hand once removes most of the mystery from everything that follows.

Explain it like I’m 5

Classification is sorting new examples into buckets based on old ones — “this new flower looks most like the ones we called setosa, so it is probably a setosa”.

The simplest classifier that works

Here is an idea so obvious it feels like cheating: to classify something new, find the example it most resembles and copy that answer. This is nearest neighbor, and it is a genuine algorithm people use.

“Most resembles” just means closest. With two features per example you can measure distance exactly as you would on a map: the difference in each direction, squared, summed, square-rooted.

Example 
import numpy as np

# Two features per flower: petal length and petal width.
X = np.array([[1.4, 0.2], [1.3, 0.2], [1.5, 0.4],
              [4.7, 1.4], [4.5, 1.5], [4.9, 1.5]])
y = np.array(["setosa", "setosa", "setosa",
              "versicolor", "versicolor", "versicolor"])

def nearest_neighbour(point):
    """Predict by finding the closest example we have already seen."""
    distances = np.sqrt(((X - point) ** 2).sum(axis=1))
    return y[distances.argmin()]

print(nearest_neighbour(np.array([1.4, 0.3])))
print(nearest_neighbour(np.array([4.6, 1.4])))

predictions = np.array([nearest_neighbour(row) for row in X])
print("accuracy on training data:", (predictions == y).mean())
Output 
setosa
versicolor
accuracy on training data: 1.0
A complete classifier, in three lines of function body.

Accuracy, and why 100% should worry you

Accuracy is the fraction of predictions that were right, measured here against the training data. It is the obvious measure and the right place to start.

But look at that result: 100% accurate. Before celebrating, notice what was measured. The classifier predicted the labels of the same six flowers it was given, and its rule is “find the closest example”, so for any training row, the closest example is itself, at distance zero. It scored perfectly by memorizing.

That number tells you nothing about a flower it has never seen, which is the only thing you actually want to know. The fix has its own lesson, immediately after this one.

Use the nearest-neighbor classifier on three flowers it has never seen. The predict function is written for you. Call it on each row of new_flowers, then count how many of the known examples it gets right.

import numpy as np

X = np.array([[1.4, 0.2], [1.3, 0.2], [4.7, 1.4], [4.5, 1.5]])
y = np.array(["setosa", "setosa", "versicolor", "versicolor"])

def predict(point):
    distances = np.sqrt(((X - point) ** 2).sum(axis=1))
    return y[distances.argmin()]

new_flowers = np.array([[1.5, 0.3], [4.6, 1.5], [1.2, 0.1]])

# TODO: print each new flower and its prediction

# TODO: count how many known examples are predicted correctly
correct = 0

print("correct on known examples:", correct, "of", len(X))

Confidence is not correctness

Most classifiers can report how sure they are, usually as a probability per class. It is a useful number and a dangerous one.

A model saying “95% confident this is spam” is reporting how strongly its learned pattern matches, not the probability that it is right. A model trained on bad data can be confidently, consistently wrong, and its confidence will not waver.

The nearest-neighbor classifier above makes this vivid: hand it a photograph of a car encoded as two numbers and it will return “setosa” with no hesitation, because setosa was the closest thing it had ever seen. A model can only answer within the world of its examples.

Common mistake: Celebrating accuracy measured on the training data

Why it happens:

It is the data in front of you, and the number comes out high.

How to fix it:

That measures memorization. Hold some examples back and score on those instead. The next lesson shows how.

Common mistake: Trusting accuracy when the classes are unbalanced

Why it happens:

90% sounds good in isolation.

How to fix it:

Work out what always guessing the commonest class would score. If that is also 90%, your model has learned nothing.

Common mistake: Treating a confidence score as a probability of being right

Why it happens:

They are both percentages and both sound authoritative.

How to fix it:

Confidence describes how well the input matched the learned pattern. On data unlike the training set it can be high and wrong.

Common mistake: Using classification where the answer is really a number

Why it happens:

Buckets feel simpler than a continuous value.

How to fix it:

Splitting price into “cheap” and “expensive” throws away information. If the answer is a number, use regression.

What does a classifier predict?

Why is 100% accuracy on the training data not impressive?

A model labels every email 'not spam' and scores 99% accuracy. What does that tell you?

Mini exercise (hard)

Build a confusion matrix by hand: the four counts that show how a classifier is wrong, not just how often. Given the true labels and the predictions for eight emails, count the true positives, true negatives, false positives and false negatives, then compute accuracy from them.

Have a go. Finish the code and press Run to see the result immediately, right on this page.

actual =    ["spam", "spam", "ok", "ok", "spam", "ok", "ok", "spam"]
predicted = ["spam", "ok",   "ok", "ok", "ok",   "ok", "spam", "spam"]

# TODO: count the four outcomes
true_pos = true_neg = false_pos = false_neg = 0

print("true positives: ", true_pos)
print("true negatives: ", true_neg)
print("false positives:", false_pos)
print("false negatives:", false_neg)
print("accuracy:", round((true_pos + true_neg) / len(actual), 2))

What to learn next

You wrote a nearest-neighbor classifier from scratch, met the X and y convention, and measured accuracy — then found the catch. Scoring 100% on the examples it memorized means nothing, and accuracy itself flatters a model whenever one class dominates.

Both of the last two lessons ended on that same warning, and now it gets fixed. Train/Test Split and Overfitting is the most important idea in this unit.