Unit 9 Project: Harden a Script You Already Wrote
Same engine, now with a dashboard and seatbelts
In this lesson
This project applies the whole unit to the script you built in Units 7 and 8: fetch records, clean them, store them. It works. What it cannot do is tell you what happened when it runs unattended, survive one bad record, or describe its own data. Four passes fix that, and you can see the difference at each step.
Explain it like I’m 5
Same engine, but now it has a dashboard, seatbelts, and a manual.
The starting point
Here is the script as it stands, written the way most working code starts out. Nothing about it is stupid, and every line of it is a liability once it runs somewhere you are not watching.
def load(raw_rows):
books = []
for row in raw_rows:
try:
books.append({"title": row[0], "year": int(row[1])})
except Exception:
pass # a bad row just disappears
print("done")
return books
Four passes over the same file
Take them one at a time, running the script after each. The order matters: logging first means every later step has a way to report itself.
- Logging. Replace
printwith leveled logs. Totals at INFO, each rejected row at WARNING. - Error handling. Replace the bare
exceptwith a custom exception raised for each specific failure, caught per row so one bad record does not end the run. - Dataclasses. Turn the dicts into a real
Booktype with named fields. - Type hints. Annotate the functions, run the checker, fix what it finds.
Build the hardened version. Convert each row into a Book, raise InvalidRow for a missing title or an unparseable year, log a WARNING for each rejection and an INFO total at the end, and return the good books.
import logging, sys
from dataclasses import dataclass
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s",
stream=sys.stdout, force=True)
RAW = [("Dune", "1965"), ("", "1984"), ("Neuromancer", "nineteen"), ("Snow Crash", "1992")]
@dataclass
class Book:
title: str
year: int
class InvalidRow(Exception):
pass
def to_book(row):
# TODO: raise InvalidRow for a missing title or an unparseable year
return Book(row[0], int(row[1]))
def load(rows):
books = []
# TODO: convert each row, warn and skip on InvalidRow, log the total
return books
for book in load(RAW):
print(book)In to_book, unpack the row, raise InvalidRow("missing title") when the title is empty, and wrap int() in a try that raises InvalidRow(f"bad year {raw_year!r}") from err. In load, catch InvalidRow per row and use logging.warning("skipping %s: %s", row, err).
import logging, sys
from dataclasses import dataclass
logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s",
stream=sys.stdout, force=True)
RAW = [("Dune", "1965"), ("", "1984"), ("Neuromancer", "nineteen"), ("Snow Crash", "1992")]
@dataclass
class Book:
title: str
year: int
class InvalidRow(Exception):
pass
def to_book(row):
title, raw_year = row
if not title:
raise InvalidRow("missing title")
try:
year = int(raw_year)
except ValueError as err:
raise InvalidRow(f"bad year {raw_year!r}") from err
return Book(title, year)
def load(rows):
books = []
for row in rows:
try:
books.append(to_book(row))
except InvalidRow as err:
logging.warning("skipping %s: %s", row, err)
logging.info("kept %d of %d rows", len(books), len(rows))
return books
for book in load(RAW):
print(book)What the annotated version looks like
The last pass adds the hints. They change nothing at runtime, and they are what let a checker verify the pieces fit: that to_book really does return a Book, and that callers of load get a list of them.
def to_book(row: tuple[str, str]) -> Book:
"""Convert one raw row into a Book, or raise InvalidRow."""
...
def load(rows: list[tuple[str, str]]) -> list[Book]:
"""Convert every row it can, logging and skipping the rest."""
...
Common mistake: Rewriting the script instead of improving it
Once you start changing things, a clean rewrite feels tempting.
Make the four passes separately and run the script after each. A rewrite loses working behavior you had forgotten was deliberate.
Common mistake: Adding logging and then never reading the log
The work felt complete once the calls were in place.
Run it and read the output. That is how you find out you are logging every row and nothing useful, or the reverse.
Common mistake: Catching InvalidRow around the whole loop rather than per row
One try around everything is less typing.
Put the try inside the loop. Outside it, the first bad row ends the entire run — exactly what the exception was meant to prevent.
Common mistake: Annotating the types while leaving the error handling alone
Hints are the most visible of the four changes.
Hints do not catch bad data, only mismatched code. A well-annotated script with except Exception: pass is still silently losing records.
Why should the try/except go inside the loop rather than around it?
Catching per row is what makes the batch resilient. Outside the loop, the first failure stops everything.
Which of the four passes would catch a caller passing a list where a tuple was expected?
That is a code-shape mismatch, which is exactly what a static checker finds.
What should the log show after a run that dropped two of four rows?
Enough to reconstruct what happened without drowning the file: the oddities individually, and the totals once.
Mini exercise (hard)
Extend the hardened loader to store its results. After building the Book list, write them into a SQLite table with parameterized inserts, log how many were stored, then read them back and confirm the round trip returned equal objects.
Give it a shot. Complete the code and press Run; there’s nothing to download or configure.
import sqlite3
from dataclasses import dataclass
@dataclass
class Book:
title: str
year: int
books = [Book("Dune", 1965), Book("Snow Crash", 1992)]
conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE books (title TEXT, year INTEGER)")
# TODO: insert each book with placeholders, then commit
loaded = [] # TODO: read the rows back as Book objects
print(len(loaded))
print(loaded == books)Reuse the Unit 8 pattern: create the table, insert (book.title, book.year) per book, commit once at the end. Rebuild with Book(*row) and compare the lists with ==.
import sqlite3
from dataclasses import dataclass
@dataclass
class Book:
title: str
year: int
books = [Book("Dune", 1965), Book("Snow Crash", 1992)]
conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE books (title TEXT, year INTEGER)")
for book in books:
conn.execute("INSERT INTO books VALUES (?, ?)", (book.title, book.year))
conn.commit()
loaded = [Book(*row) for row in conn.execute("SELECT title, year FROM books")]
print(len(loaded))
print(loaded == books)