Continuous Integration with GitHub Actions

In this lesson

You have tests. The problem with tests is remembering to run them, and specifically remembering on the day you are in a hurry — which is the day you break something.

Continuous integration is the fix: a server runs your tests automatically on every push, on a clean machine, and tells you within a minute if anything broke.

Explain it like I’m 5

CI is a robot that checks out your code on a fresh computer, runs your tests, and tells everyone whether it worked.

What CI actually catches

“It runs my tests, and I can do that myself” undersells it considerably. The value is in the word clean. The CI machine has none of your local state, so it catches a specific and very common class of problem:

A package you installed months ago and never added to dependencies.
A file that only exists on your machine, or is in .gitignore and never got committed.
A test that passes only because an earlier test left something behind.
Code that works on your Python version and not on the one your users have.
The commit where you meant to run the tests and did not.

Every one of those is invisible locally by definition. That is why “works on my machine” is a joke with a very long life.

The workflow file, line by line

GitHub Actions looks for YAML files in .github/workflows/. One file is a workflow; it contains jobs, and each job is a list of steps run on a fresh virtual machine.

That is the entire model. Everything else is detail.

Example · .github/workflows/tests.yml

name: tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install the project and its dev extras
        run: pip install -e ".[dev]"

      - name: Run the tests
        run: pytest -q

A complete, working CI setup. Twenty lines, and it never forgets.

What this code does

name is what appears in the Actions tab. on is the trigger: here, any push to main and every pull request. Running on pull requests is the more valuable half: it checks the code before it is merged.

runs-on: ubuntu-latest asks GitHub for a fresh Linux machine, thrown away when the job ends. That freshness is the whole point.

uses: runs a pre-built action someone else wrote. actions/checkout@v4 clones your repository. Without it the machine has no code at all. actions/setup-python@v5 installs the version you name. Pin the @v4/@v5; unpinned actions change under you.

run: is a shell command. pip install -e ".[dev]" installs your package from the pyproject.toml you wrote last lesson, plus the dev extras, which is how pytest gets onto the machine. If a dependency is missing from that file, this is the step that fails, and that is a bug worth finding.

Any step exiting non-zero fails the job and stops the rest. pytest exits 1 when a test fails, so the wiring needs nothing else.

Testing several Python versions at once

A matrix runs the same job several times with different values substituted in. The usual use is checking that your code works on every Python version you claim to support.

Note that this multiplies your minutes. Three versions is three jobs. Match it to the requires-python in your pyproject.toml and no wider.

Example · .github/workflows/tests.yml

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - run: pip install -e ".[dev]"
      - run: ruff check .
      - run: pytest -q

Three jobs from one definition, plus a lint step.

Read a workflow the way GitHub does: as data. Parse it with yaml.safe_load, then work out how many jobs the matrix expands to and what each one will be called. Watch what happens to the on: key; the answer is genuinely surprising and it catches people out for real.

import yaml

WORKFLOW = """
name: tests
on:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - run: pytest -q
"""

data = yaml.safe_load(WORKFLOW)

# TODO: the top-level keys, exactly as PyYAML produced them
keys = []

# TODO: the versions in the matrix, and the name of each job GitHub will run
versions = []
job_names = []

print("top-level keys:", keys)
print("is 'on' a string key?", "on" in data)
print("jobs:", job_names)

import yaml

WORKFLOW = """
name: tests
on:
  push:
    branches: [main]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - run: pytest -q
"""

data = yaml.safe_load(WORKFLOW)

keys = list(data)

versions = data["jobs"]["test"]["strategy"]["matrix"]["python-version"]
job_names = [f"test ({version})" for version in versions]

print("top-level keys:", keys)
print("is 'on' a string key?", "on" in data)
print("jobs:", job_names)

Reading a failing run

A red cross in the Actions tab is a traceback in a different outfit, and the same playbook applies: find the first thing that actually failed, not the last thing printed.

Open the failed job, expand the step marked with the red cross, and scroll to its first error line. Later steps are usually cascading consequences.

The three failures you will actually see, in order of frequency:

ModuleNotFoundError in the install step — a dependency you have locally and did not declare. Fix pyproject.toml, not the workflow.
A test that passes locally and fails on CI — nearly always a missing file, a hard-coded path, or a test that depended on another test's leftovers.
One matrix job failing — you used syntax newer than the oldest Python you claim to support.

Example

Run pytest -q
..F..                                                      [100%]
=================================== FAILURES ===================================
____________________________ test_report_from_file ____________________________

    def test_report_from_file():
>       rows = load("data/sales.csv")
E       FileNotFoundError: [Errno 2] No such file or directory: 'data/sales.csv'

tests/test_report.py:18: FileNotFoundError
=========================== short test summary info ============================
FAILED tests/test_report.py::test_report_from_file - FileNotFoundError
1 failed, 4 passed in 0.34s
Error: Process completed with exit code 1.

The classic first CI failure: a file that only exists on your machine.

What else to put in the pipeline

Once tests run on every push, adding a check costs one line. The ones worth having, in the order they should run:

Lint — ruff check .. Two seconds, catches unused imports and undefined names.
Format check — ruff format --check .. Fails if anyone committed unformatted code, which ends the discussion for good.
Tests — pytest -q.
Types — mypy src/, if you added the type hints from Unit 9.
Build — python -m build. Proves the package from the last lesson still builds, before a user discovers it does not.

Resist the urge to add more. A CI run that takes fifteen minutes gets ignored, and an ignored pipeline is worse than none — it is a green tick nobody reads.

Common mistake: Setting up CI before writing any tests

Why it happens:

The workflow file is the visible, satisfying part.

How to fix it:

A pipeline with nothing to run reports success unconditionally, which is worse than no pipeline. Tests first.

Common mistake: Fixing a CI failure by changing the workflow

Why it happens:

The red cross is on the workflow page, so the workflow looks like the broken thing.

How to fix it:

Usually CI is right and your project is wrong. A ModuleNotFoundError means a genuinely missing dependency. Installing it in an extra step hides the bug your users will hit.

Common mistake: Leaving fail-fast on for a matrix

Why it happens:

It is the default, so it takes a decision to change it.

How to fix it:

Canceling the other jobs throws away the most useful information you had: whether one Python version broke or all of them.

Common mistake: Adding so many checks that nobody waits for the result

Why it happens:

Each individual check seems worth having.

How to fix it:

Keep the pipeline under a couple of minutes. Lint, test, build. Anything slower belongs on a nightly schedule, not on every push.

Common mistake: Putting secrets straight into the workflow file

Why it happens:

The step needs an API key and the file is right there.

How to fix it:

Workflow files are committed, so the key is now public and permanently in the history. Use repository secrets and read them with ${{ secrets.NAME }}, exactly as Unit 7 argued for environment variables.

What does a CI run catch that running tests locally does not?

What does actions/checkout@v4 do?

A step failed with ModuleNotFoundError during pip install. Where is the bug?

Why does the YAML key on: parse as True?

Mini exercise (medium)

Write the thing you will actually reach for when a run goes red: a reader for CI logs. first_failure(log) takes the log as text and returns (step_name, error_line) for the first step that failed. Later failures are usually consequences. Steps start with "Run ", and a failure is a line containing "Error:". Return None when nothing failed.

Try it yourself. Complete the snippet and hit Run; it executes in your browser, no setup required.

LOG = """Run actions/checkout@v4
Syncing repository: ada/tidyup
Run pip install -e ".[dev]"
Collecting rich>=13.0
Could not find a version that satisfies the requirement rich>=13.0
Error: Process completed with exit code 1.
Run pytest -q
Error: Process completed with exit code 2.
"""

CLEAN = """Run actions/checkout@v4
Run pytest -q
5 passed in 0.31s
"""

def first_failure(log):
    """(step name, error line) for the FIRST failing step, or None."""
    # TODO: remember the current step; a step line starts with "Run "
    # TODO: return as soon as a line contains "Error:"
    return None

found = first_failure(LOG)
print("step: ", found[0])
print("error:", found[1])
print("clean log:", first_failure(CLEAN))

LOG = """Run actions/checkout@v4
Syncing repository: ada/tidyup
Run pip install -e ".[dev]"
Collecting rich>=13.0
Could not find a version that satisfies the requirement rich>=13.0
Error: Process completed with exit code 1.
Run pytest -q
Error: Process completed with exit code 2.
"""

CLEAN = """Run actions/checkout@v4
Run pytest -q
5 passed in 0.31s
"""

def first_failure(log):
    """(step name, error line) for the FIRST failing step, or None."""
    step = None
    for line in log.splitlines():
        if line.startswith("Run "):
            step = line[4:]
        elif "Error:" in line:
            return step, line.strip()
    return None

found = first_failure(LOG)
print("step: ", found[0])
print("error:", found[1])
print("clean log:", first_failure(CLEAN))

What to learn next

You set up a robot that checks your work: a workflow file with triggers, jobs and steps, actions/checkout and setup-python, a matrix that runs three Python versions at once with fail-fast turned off, and the reading habit for a red run: first failing step, not the last error. You also met the YAML on: trap and saw it parse as True with your own eyes.

CI proves your code works on a clean Linux machine. Containers in Plain English is the next step out: packaging the operating system and the Python version along with your code, so “a clean machine” becomes something you define rather than hope for.