Waiting vs Working: Why Programs Are Slow

The one question that decides which tool helps

Advanced 12 min

In this lesson

The script you built across Units 7 to 10 is careful, logged, and free of repetition. It is also slow — and not because the code is bad. It is slow because it spends almost all of its life doing nothing at all, sitting still while a server somewhere thinks about its answer.

This unit fixes that, using concurrency. But before touching a single tool, you need the distinction that decides which tool can help, and whether any of them can: is your program working, or is it waiting?

Explain it like I’m 5

If you are waiting for a kettle to boil, you can chop vegetables while it does. If you have a mountain of vegetables and no kettle, waiting cleverly does not help you at all. You need more hands.

Two completely different kinds of slow

Programs are slow for one of two reasons, and the fix for each is the opposite of the fix for the other.

I/O-bound work is waiting on something outside Python: a network request to an API, a database query, reading a big file from disk. Your CPU is idle. The word I/O just means input/output: anything that leaves your program and comes back.

CPU-bound work is Python genuinely working: resizing images, crunching numbers, parsing a huge file that is already in memory. Your CPU is flat out.

Here is the test, and it is the whole lesson in one sentence: if you froze the program mid-run, would it be in the middle of a calculation, or sitting on its hands?

Example 
import time

def fetch_page(number):
    """Stand-in for a real network call: almost entirely waiting."""
    time.sleep(0.2)               # the server is thinking; Python is idle
    return f"page {number}"

start = time.perf_counter()
pages = [fetch_page(n) for n in range(10)]
elapsed = time.perf_counter() - start

print(f"fetched {len(pages)} pages in {elapsed:.1f}s")
Output 
fetched 10 pages in 2.0s
Ten calls, two seconds. Almost none of it is your program.

Why waiting is the fixable kind

Ten calls that each wait 0.2 seconds take 2 seconds in a row. But nothing forces them to be in a row. If you start all ten and then collect the answers, the waiting overlaps: you spend 0.2 seconds waiting for all of them at once instead of 0.2 seconds ten times.

Working does not overlap like that. Two seconds of real calculation is two seconds of calculation no matter how you arrange it, unless you genuinely bring more processors to bear. That asymmetry is the reason this unit exists, and the reason its tools help dramatically in some situations and not at all in others.

Work out what the overlap is worth before you write any concurrent code. Each call waits WAIT seconds and then does WORK seconds of real processing. Running them one at a time costs the full total. Running WORKERS at once means the waiting is shared (the calls split into ceil(CALLS / WORKERS) rounds), but the working still has to happen for every call. Fill in concurrent.

import math

WAIT, WORK = 0.2, 0.01      # seconds per call: waiting, then working
CALLS, WORKERS = 10, 5

sequential = CALLS * (WAIT + WORK)

# TODO: the waiting splits into rounds; the working does not overlap at all
concurrent = 0

print(f"sequential: {sequential:.2f}s")
print(f"concurrent: {concurrent:.2f}s")
print(f"speedup:    {sequential / concurrent:.1f}x")

Notice what the speedup was not

Five workers gave 4.2x, not 5x. The missing fraction is the WORK part, the piece that cannot overlap. Push WORK up and the speedup collapses towards 1x no matter how many workers you add. Push it down and you approach the worker count.

That single line of arithmetic predicts everything this unit does. Concurrency divides your waiting by the number of workers, and leaves your working exactly where it was.

Three things concurrency will not do

Before you reach for it, know what it does not buy.

It will not make one call faster. One request that takes 2 seconds still takes 2 seconds. Concurrency only helps when there are several things to wait for.

It will not speed up a CPU-bound loop — not with threads, anyway. Adding threads to a calculation in Python usually produces exactly the same runtime, sometimes slightly worse. The reason is a lock inside Python called the GIL, and it surprises everyone; you will meet it properly in The GIL, Processes, and Shared State.

It will not pay for itself on a script that was never slow. Concurrency makes code harder to read, harder to debug, and capable of failing in ways sequential code cannot. If your script finishes in under a second, the correct amount of concurrency is none.

Common mistake: Adding threads to a CPU-bound loop and seeing no gain

Why it happens:

Slow is slow, and concurrency is advertised as the cure for slow.

How to fix it:

Ask the working-or-waiting question first. Calculation needs separate processes, not threads. The next few lessons explain why.

Common mistake: Expecting one request to get faster

Why it happens:

“Faster program” and “faster request” sound like the same thing.

How to fix it:

Concurrency overlaps separate waits. With only one thing to wait for there is nothing to overlap, and the wait is whatever the server decides it is.

Common mistake: Reaching for concurrency before measuring

Why it happens:

It feels like the professional answer, and rewriting is more fun than measuring.

How to fix it:

Time the thing first, with time.perf_counter() or the @timer from Unit 10. Often the real cost turns out to be one accidental query in a loop.

Common mistake: Making a fast script concurrent anyway

Why it happens:

The technique is newly learned, so everything looks like a candidate.

How to fix it:

Concurrency has a permanent readability and debugging cost. Spend it where there is real waiting to reclaim, and nowhere else.

Is downloading fifty files I/O-bound or CPU-bound?

Does running things concurrently make a single request faster?

Five workers on ten calls gave 4.2x rather than 5x. Why?

Mini exercise (easy)

Classify eight real jobs. For each one decide whether the program is mostly waiting ("io") or mostly working ("cpu"), and let the supplied tool_for() name the tool that suits it. Getting the classification right is the whole skill; the tool follows automatically.

Now you. Edit the starter code below, then Run it, everything happens in the browser.

# Mark each job "io" if the program is mostly WAITING,
# or "cpu" if it is mostly WORKING.
JOBS = {
    "download 200 product pages": "?",
    "resize 500 photos": "?",
    "run 1000 database queries": "?",
    "find every prime below 10 million": "?",
    "upload 50 files to cloud storage": "?",
    "compress a 4 GB video": "?",
    "call a weather API for 30 cities": "?",
    "sort a list of 20 million numbers": "?",
}

def tool_for(kind):
    """The tool that suits each kind of work."""
    return "threads or async" if kind == "io" else "processes"

for job, kind in JOBS.items():
    print(f"{kind}: {job} -> {tool_for(kind)}")

What to learn next

You learned the question that decides everything in this unit: is the program working or waiting? I/O-bound work is waiting and can be overlapped; CPU-bound work is real processing and cannot. You also did the arithmetic showing why five workers gave 4.2x rather than 5x, and saw the three things concurrency will never do for you.

Now collect the win. Doing Many Things at Once with Thread Pools turns those two seconds into four tenths, in about four lines.