Doing Many Things at Once with Thread Pools
A small team of assistants running your errands
In this lesson
You now know that waiting is the fixable kind of slow. This lesson is the most practical way to fix it. ThreadPoolExecutor takes a list of jobs and a small team of workers, runs the jobs concurrently, and hands you the results, usually in about four lines.
Python does have a lower-level threading module. You will almost never want it. The pool handles starting, finishing, and cleaning up the workers, which is where hand-rolled threading code goes wrong.
Explain it like I’m 5
A thread pool is a small team of assistants. You hand out the errands, they run them at the same time, and you collect the results as they come back, in whatever order they happen to finish.
The four-line version
ThreadPoolExecutor is a context manager, the Unit 10 tool. The with block guarantees the workers are shut down properly even if something fails inside it.
pool.map() works like the built-in map(): give it a function and a list of inputs, get back the results in input order. It is the right first choice, and often the only thing you need.
import time
from concurrent.futures import ThreadPoolExecutor
def fetch_page(number):
time.sleep(0.2) # the network, waiting
return f"page {number}"
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=5) as pool:
pages = list(pool.map(fetch_page, range(10)))
print(f"fetched {len(pages)} pages in {time.perf_counter() - start:.1f}s")
print(pages[:3])
fetched 10 pages in 0.4s ['page 0', 'page 1', 'page 2']
Results really do arrive out of order
map hides something you need to see, because it will bite you the moment you stop using it. The workers finish whenever they finish. A fast page overtakes a slow one. map quietly holds the results and gives them back in the order you submitted, but the completion order is genuinely scrambled.
When you want results as soon as they are ready, use submit() instead. It hands back a future, a receipt for a result that does not exist yet, and as_completed() yields those futures in the order they actually finish.
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
def fetch_page(number):
time.sleep(0.5 - number * 0.1) # later pages happen to be quicker
return f"page {number}"
with ThreadPoolExecutor(max_workers=5) as pool:
futures = [pool.submit(fetch_page, n) for n in range(5)]
for future in as_completed(futures):
print("arrived:", future.result())
arrived: page 4 arrived: page 3 arrived: page 2 arrived: page 1 arrived: page 0
Predicting what a pool will buy you
Workers do not sit in neat rounds when the jobs are different lengths. Each worker takes the next job the moment it is free, so the finish time is decided by whichever worker ends up with the heaviest pile. Being able to estimate that tells you whether a pool is worth adding at all.
Model the pool. Each worker starts free at time 0; every job goes to whichever worker becomes free soonest, and the total wall-clock time is when the last worker finishes. Fill in wall_clock so it returns that. With one worker it should simply add everything up.
DURATIONS = [3, 1, 2, 1, 4, 1] # seconds each job spends waiting
def wall_clock(durations, workers):
ends = [0] * workers # when each worker becomes free
# TODO: give each job to the worker that is free soonest,
# then return the moment the last one finishes
return 0
print("one at a time:", wall_clock(DURATIONS, 1))
print("four workers: ", wall_clock(DURATIONS, 4))Loop over the durations. Each time, find the smallest value in ends with ends.index(min(ends)) (that is your free worker) and add the duration to it. When the loop is done, the answer is max(ends), because you are not finished until the slowest worker is.
DURATIONS = [3, 1, 2, 1, 4, 1] # seconds each job spends waiting
def wall_clock(durations, workers):
ends = [0] * workers # when each worker becomes free
for duration in durations:
soonest = ends.index(min(ends))
ends[soonest] += duration
return max(ends)
print("one at a time:", wall_clock(DURATIONS, 1))
print("four workers: ", wall_clock(DURATIONS, 4))How many workers?
More is not better. Each thread costs memory, and every extra one aimed at the same server makes you a worse citizen of the internet. A hundred simultaneous requests is indistinguishable from an attack, and you will be rate-limited or blocked.
For network work, start between 5 and 20 and measure. The gain flattens quickly: the jump from 1 to 5 workers is transformative, 5 to 10 is noticeable, 10 to 50 is usually nothing at all because the bottleneck has moved to the far end. Whatever the API's documented rate limit is, that is your real ceiling. The Unit 11 project puts a proper bound on it.
Common mistake: Assuming results come back in submission order
map reorders them for you, so the scrambling stays invisible until you switch to submit.
Use map when order matters, or submit a {future: input} dict so every result can be matched back to what produced it.
Common mistake: Using a pool of two hundred
If five workers helped, four hundred should help more.
The gain flattens fast and the target server starts refusing you. Stay in the 5–20 range for network work and respect the documented rate limit.
Common mistake: Reaching for threads to speed up number crunching
The pool made the fetching dramatically faster, so it looks like a general speed tool.
Threads overlap waiting, not working. Use ProcessPoolExecutor for calculation.
Common mistake: Losing exceptions raised inside workers
A failing job does not crash the pool; the error waits quietly inside its future.
The exception surfaces when you call future.result(). If you never collect the results, you never learn it failed.
What does executor.map return?
map re-sorts for you. Use submit plus as_completed when you want them as they land.
With submit and as_completed, do results arrive in submission order?
That is the entire purpose of as_completed: hand back each result the moment it is ready.
What kind of work suits a thread pool?
Threads overlap waiting. Calculation needs processes instead.
Mini exercise (medium)
Five pages were submitted in order 0 to 4 but finished scrambled. Each arrival is a (submission_index, value) pair. Write completion_order() to return the values as they actually arrived, and in_submission_order() to put them back the way they were submitted, which is the work map does invisibly on your behalf.
Try it yourself. Complete the snippet and hit Run; it executes in your browser, no setup required.
# Five pages were submitted in order 0..4 but finished scrambled.
# Each arrival is (submission_index, value).
ARRIVALS = [(2, "page 2"), (0, "page 0"), (4, "page 4"), (1, "page 1"), (3, "page 3")]
def completion_order(arrivals):
return [] # TODO: just the values, in the order they arrived
def in_submission_order(arrivals):
return [] # TODO: the values, back in the order they were submitted
print("as they arrived:", completion_order(ARRIVALS))
print("in order: ", in_submission_order(ARRIVALS))completion_order is just the second item of each pair, left alone. For in_submission_order, sort the pairs by their first item before pulling the values out. sorted(arrivals) is enough, since tuples sort by their first element.
# Five pages were submitted in order 0..4 but finished scrambled.
# Each arrival is (submission_index, value).
ARRIVALS = [(2, "page 2"), (0, "page 0"), (4, "page 4"), (1, "page 1"), (3, "page 3")]
def completion_order(arrivals):
return [value for _, value in arrivals]
def in_submission_order(arrivals):
return [value for _, value in sorted(arrivals)]
print("as they arrived:", completion_order(ARRIVALS))
print("in order: ", in_submission_order(ARRIVALS))