Unit 11 Project: Fetch Many Pages Politely

Fast, bounded, and still a good citizen

Advanced 16 min

In this lesson

The paginated fetch you wrote in Unit 7 walks through pages one at a time. Twenty pages at half a second each is ten seconds of almost pure waiting, the exact shape of problem this unit exists to solve.

This project speeds it up without turning you into the kind of client that gets blocked. That balance is the actual work: making it fast is four lines, and making it fast and polite and debuggable is the rest of the lesson.

Explain it like I’m 5

Sending five polite runners to fetch things instead of one, and making sure all five never hammer on the door at the same moment.

Why not just fetch all twenty at once

Your instinct after the last three lessons is to launch every page simultaneously. Do that and you learn about HTTP 429, Too Many Requests. Most APIs publish a rate limit, and exceeding it gets you throttled, temporarily banned, or permanently blocked.

So the number you want is not “as many as possible.” It is the largest number that stays comfortably under the limit, which for a documented cap of 10 requests per second might be a pool of 5. Bounding concurrency is not a compromise on speed; it is the difference between a script that works and one that gets you blocked.

Decide the bound with arithmetic rather than a guess. Twelve pages, half a second each. With at_once running together, the pages go out in rounds of that size, so the total is the number of rounds multiplied by the time per page. Fill in wall_clock and compare three settings.

import math

PAGES, PER_PAGE = 12, 0.5

def wall_clock(pages, per_page, at_once):
    # TODO: pages go out in rounds of `at_once`; return the total seconds
    return 0

for at_once in (1, 4, 12):
    print(f"{at_once:2d} at once: {wall_clock(PAGES, PER_PAGE, at_once):.1f}s")

Reading those numbers properly

Going from 1 to 4 saves 4.5 seconds. Going from 4 to 12 saves another 1 second — a quarter of the benefit, for triple the load on someone else's server. This is the shape of nearly every concurrency decision you will make: the first few workers buy almost all of the win, and the rest buy trouble.

Pick the knee of that curve, not the end of it. Four is the right answer here.

The script

Every piece here comes from a previous unit. The logging is Unit 9, the @retry is Unit 10, the session and rate limiting are Unit 7. The only new part is the pool.

Example · fetch_pages.py 
import logging, time
from concurrent.futures import ThreadPoolExecutor, as_completed

import requests

logging.basicConfig(level=logging.INFO,
                    format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)

AT_ONCE = 4
PAGES = range(1, 13)


def fetch_page(session, number):
    """Fetch one page. Raises on failure; the caller decides what that means."""
    log.info("page %d: requesting", number)
    response = session.get("https://example.com/api/items",
                           params={"page": number}, timeout=10)
    response.raise_for_status()
    log.info("page %d: done", number)
    return response.json()["items"]


def fetch_all(pages, at_once=AT_ONCE):
    items, failed = [], []

    with requests.Session() as session:
        with ThreadPoolExecutor(max_workers=at_once) as pool:
            futures = {pool.submit(fetch_page, session, n): n for n in pages}

            for future in as_completed(futures):
                number = futures[future]
                try:
                    items.extend(future.result())
                except requests.RequestException as err:
                    log.warning("page %d failed: %s", number, err)
                    failed.append(number)

    return items, failed


if __name__ == "__main__":
    start = time.perf_counter()
    items, failed = fetch_all(PAGES)
    log.info("%d items from %d pages in %.1fs (%d failed)",
             len(items), len(PAGES), time.perf_counter() - start, len(failed))
Output 
2026-07-28 10:14:02 INFO page 1: requesting
2026-07-28 10:14:02 INFO page 2: requesting
2026-07-28 10:14:02 INFO page 3: requesting
2026-07-28 10:14:02 INFO page 4: requesting
2026-07-28 10:14:03 INFO page 2: done
2026-07-28 10:14:03 INFO page 5: requesting
2026-07-28 10:14:03 INFO page 1: done
...
2026-07-28 10:14:04 WARNING page 9 failed: 503 Server Error
...
2026-07-28 10:14:05 INFO 220 items from 12 pages in 1.7s (1 failed)
Four in flight at all times. One page failed and the run continued.

Why the logs matter more than they did before

Look at the output again: page 2 finishes before page 1, and page 5 starts the instant a slot frees up. Interleaved output is not a defect — it is what concurrency looks like from the outside. But it does mean a bare print("done") is now useless, because you cannot tell which page it refers to.

Two habits make interleaved output readable. Put the identifier in every message: page %d, not done. And use timestamps, which logging gives you for free and which are the only way to reconstruct what overlapped with what.

There is a practical reason to prefer logging over print here beyond the formatting: log calls are safe to make from several threads at once, and will not interleave halfway through a line. Two threads printing at the same moment can produce genuinely mangled output.

Where to take it next

Three extensions, in increasing order of difficulty:

  • Retry the failures. You have @retry from Unit 10. A 503 is worth another attempt; a 404 is not. Retry only what can plausibly succeed.
  • Stream the results. items.extend(...) holds everything in memory. For a large crawl, write each page to a file or the Unit 8 database as it arrives.
  • Rewrite it with async. Swap requests for httpx, the pool for asyncio.gather, and bound it with an asyncio.Semaphore(4) instead of max_workers. Same structure, same result, and good practice for reading async code in the wild.

Common mistake: Unbounded concurrency triggering a 429

Why it happens:

If four workers are fast, forty should be faster, and locally, against a fake server, they are.

How to fix it:

Set max_workers deliberately from the API's documented rate limit, and stay well below it.

Common mistake: One failed page aborting the whole run

Why it happens:

The try ends up around the entire loop instead of around one future.result().

How to fix it:

Catch per page, inside the loop. Log the failure, record the page number, and let the rest finish.

Common mistake: Printing without saying which page

Why it happens:

Sequential code has an obvious current item, so bare messages were always enough.

How to fix it:

Concurrent output is interleaved. Put the page number in every message and let logging add the timestamp.

Common mistake: Creating a new Session inside every worker

Why it happens:

Sharing one across threads feels like it must be unsafe.

How to fix it:

A requests.Session is fine to share here, and reusing its connection pool is a large part of the speedup. Create one and pass it in.

Why bound the number of simultaneous requests?

What should happen when one page fails?

Why is logging more useful than printing in a concurrent run?

Mini exercise (medium)

The run finished and some pages failed. A failed page arrived as None instead of its content. Write collect() to split the arrivals into the pages that worked and the numbers of the ones that did not, so the run can report honestly instead of pretending it fetched everything.

Practice here. Fill in the missing piece and click Run to try your answer in place.

# Pages as they came back. A page that failed arrived as None.
ARRIVALS = [(0, "page 0"), (1, None), (2, "page 2"), (3, None), (4, "page 4")]

def collect(arrivals):
    kept, failed = [], []
    # TODO: keep the pages that worked; record the numbers of those that did not
    return kept, failed

kept, failed = collect(ARRIVALS)
print("kept:  ", kept)
print("failed:", failed)
print(f"{len(kept)} of {len(ARRIVALS)} pages")

What to learn next

You made the paginated fetch concurrent and kept it polite: a deliberate bound taken from the rate limit rather than a guess, a try around each single future.result() so one dead page cannot end the run, a shared Session for connection reuse, and logs that identify themselves because interleaved output is no longer readable otherwise.

That completes Unit 11. Your script now fetches, cleans, stores, logs, retries, and does it several times faster. The obvious next question is what to do with all that collected data, which is where Unit 12 starts, with NumPy, pandas, and your first real analysis. When you’re ready, keep building.