Unit 13 Project: First Tiny ML Model

Split, train, predict, evaluate, and say what it cannot do

Advanced 18 min

In this lesson

Everything in this unit, applied once, properly. A small flower dataset with two features, a real classifier, an honest evaluation, and a written account of what the model cannot be trusted to do.

That last part is the deliverable people skip, and it is the one that distinguishes a project from a demo.

Explain it like I’m 5

Your first ML project is a small science experiment: set up examples, train, test on questions the model has not seen, and say plainly what you learned and what you did not.

The whole project

Five steps. Every one of them uses something from this unit, and the order is not negotiable: the split happens before training, or the evaluation is worthless.

Example · first_model.py 
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
import matplotlib.pyplot as plt

# 1. Features and labels
X = np.array([[1.4, 0.2], [1.3, 0.2], [1.5, 0.4], [1.7, 0.3],
              [4.7, 1.4], [4.5, 1.5], [4.9, 1.5], [4.6, 1.3]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])
names = ["setosa", "versicolor"]

# 2. Split - random_state makes the split repeatable
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)
print("training rows:", len(X_train), "| test rows:", len(X_test))

# 3. Train
model = DecisionTreeClassifier(max_depth=2, random_state=42)
model.fit(X_train, y_train)

# 4. Predict and evaluate
predictions = model.predict(X_test)
print("predicted:", [names[p] for p in predictions])
print("actual:   ", [names[a] for a in y_test])
print("accuracy: ", round(float(model.score(X_test, y_test)), 2))

# 5. Chart
plt.figure(figsize=(6, 3.5))
for label, color in [(0, "#2b7cd3"), (1, "#d3552b")]:
    rows = X[y == label]
    plt.scatter(rows[:, 0], rows[:, 1], color=color, label=names[label])
plt.title("Petal length vs width")
plt.xlabel("petal length")
plt.ylabel("petal width")
plt.legend()
plt.show()
Output 
training rows: 6 | test rows: 2
predicted: ['setosa', 'versicolor']
actual:    ['setosa', 'versicolor']
accuracy:  1.0
Split, train, predict, evaluate, chart.

Writing what it cannot do

The code produced a number. The project produces a paragraph, and it should cover three things: what the model does, how well and on what evidence, and where it should not be used.

For this model:

This model tells setosa from versicolor irises using petal length and width. On the two held-back examples it was correct both times, but two test rows is far too few to claim an accuracy figure. The honest statement is that it works on data resembling these eight flowers. The two species separate cleanly on these features, so the task is easy; a harder pair of species would need more data and probably more features. It has never seen a third species and will confidently mislabel one as whichever of the two it resembles more. It should not be used to identify real flowers.

Notice that almost all of it is limits. A model's usefulness is bounded by the data it saw, and saying so plainly is not modesty — it is the accurate description.

Run the core of the project yourself: split the data with test_size=0.25 and random_state=42, train a DecisionTreeClassifier with max_depth=2 and random_state=42, then report the accuracy on the held-back rows and the predictions it made.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X = np.array([[1.4, 0.2], [1.3, 0.2], [1.5, 0.4], [1.7, 0.3],
              [4.7, 1.4], [4.5, 1.5], [4.9, 1.5], [4.6, 1.3]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])

# TODO: split off 25% for testing, with random_state=42
X_train, X_test, y_train, y_test = X, X, y, y

# TODO: create and fit a DecisionTreeClassifier(max_depth=2, random_state=42)
model = None

print("test accuracy:", 0)
print("predictions:", [])

Where to go next

You now have the workflow that every supervised-learning project shares. Three honest next steps:

  • Use a real dataset. from sklearn.datasets import load_iris gives you 150 flowers and three species, enough that the accuracy figure starts to mean something.
  • Try several models in a loop. The shared interface makes this a few lines, and comparing a tree against a linear model on the same split teaches more than either alone.
  • Learn cross-validation. With small data, one split is luck. cross_val_score repeats the split several ways and averages, which is the standard answer to “was that score a fluke?”

And the honest framing for what you have learned: this unit covered the workflow and the skepticism, which is most of what makes machine learning useful in practice. It did not cover the math behind the algorithms, choosing between dozens of models, or what happens when data does not fit in memory. Those are real subjects, but they build on exactly the four steps you now know.

Common mistake: Skipping evaluation entirely

Why it happens:

The predictions look plausible, so the model seems fine.

How to fix it:

Always score on held-back data. Plausible-looking predictions are exactly what a broken model produces.

Common mistake: Splitting sorted data without shuffling

Why it happens:

Slicing the last rows off is the obvious way to hold data back.

How to fix it:

If the rows are grouped by label, the test set ends up all one class. train_test_split shuffles for you.

Common mistake: Calling the model production-ready

Why it happens:

A high accuracy number feels like a finished result.

How to fix it:

State the size of the test set and the range of data it saw. Two test rows is a demonstration, not evidence.

Common mistake: Letting a tree grow without a depth limit

Why it happens:

The default settings train without complaint and score perfectly.

How to fix it:

An unbounded tree memorizes. Set max_depth and compare the training and test scores.

What are the project steps, in order?

Why explain a model's limitations?

What should happen before predictions are trusted?

Mini exercise (hard)

Put the project's own warning to the test. Train two decision trees on the same split, one with max_depth=1 and one with no limit at all, and print each one's training accuracy, test accuracy, and the gap between them. Then print whether the two gaps actually differ, which is the question that decides if this test set can tell a careful model from a memorizing one.

Have a go. Finish the code and press Run to see the result immediately, right on this page.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier

X = np.array([[1.4, 0.2], [1.3, 0.2], [1.5, 0.4], [1.7, 0.3],
              [4.7, 1.4], [4.5, 1.5], [4.9, 1.5], [4.6, 1.3]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)

gaps = {}
# TODO: for each depth, fit the tree and print train accuracy, test accuracy,
#       and the gap between them - recording the gap in `gaps`
for name, depth in [("depth 1  ", 1), ("unlimited", None)]:
    pass

print("gaps:", gaps)
# TODO: do the two gaps actually differ?
print("test set can tell them apart:", None)

What to learn next

You ran a complete supervised-learning project: features and labels, a shuffled train/test split, a depth-limited tree, a score measured only on held-back rows, a chart, and a written statement of what the model cannot do. The exercise then showed the sharpest lesson of all: with two test rows, the evaluation could not tell a careful model from a memorizing one.

That completes Unit 13. You have spent six units at the keyboard building things that run in a terminal. Unit 14 is a change of pace: interactive programs (games and windows) and the event-loop thinking they demand. When you’re ready, keep building.