Generators and the yield Keyword
Produce values one at a time with yield
In this lesson
A generator produces values one at a time, only as you ask for them — using the yield keyword instead of return — rather than building a whole list in memory at once. That makes generators ideal for scripts that stream through big files or long sequences without loading everything into memory. You met generator expressions briefly; here you’ll write full generator functions and see exactly how yield works.
Explain it like I’m 5
A normal function hands you a finished basket of fruit. A generator hands you one piece each time you reach out — it pauses in between and only picks the next piece when you ask.
From return to yield
Put yield in a function and it becomes a generator function. Where return ends a function and hands back one value, yield hands back a value and pauses the function in place — keeping all its variables. The next time you ask for a value, it resumes right after the yield, remembering exactly where it was.
def count_up_to(limit):
n = 1
while n <= limit:
yield n
n += 1
for number in count_up_to(3):
print(number)
1 2 3
Lazy by default, and one-shot
def announce(items):
print("generator starting")
for item in items:
yield item
gen = announce(["a", "b"])
print("made the generator, but nothing has run yet")
print(next(gen))
print(next(gen))
made the generator, but nothing has run yet generator starting a b
def squares(nums):
for n in nums:
yield n * n
gen = squares([1, 2, 3])
print(next(gen))
print(next(gen))
print(list(gen))
print(list(gen))
1 4 [9] []
The walrus operator := (assignment expression)
Python 3.8 added the walrus operator :=, also called an assignment expression. It assigns a value and returns it in the same step, which lets you write certain loops more cleanly.
The classic use case is a while loop that reads data in chunks: you want to read, check whether you got anything, and use the result — all without calling .read() twice or using a separate assignment line.
# Without := you need two lines or a read-twice pattern.
chunk = f.read(1024)
while chunk:
process(chunk)
chunk = f.read(1024)
# With := -- read, assign, and test in one expression.
while chunk := f.read(1024):
process(chunk)
# := works in any expression context, not just while loops.
data = [1, 2, 3, 4, 5, 6]
# Process pairs until fewer than 2 items remain.
while (n := len(data)) >= 2:
a, b = data.pop(0), data.pop(0)
print(f"pair: {a}, {b} (remaining: {n - 2})")
pair: 1, 2 (remaining: 4) pair: 3, 4 (remaining: 2) pair: 5, 6 (remaining: 0)
The real win: stream data without loading it all
This is where generators earn their keep in real scripts. Python’s file objects are already lazy — looping path.open() line by line never loads more than one line at a time. A generator wrapping that lets you filter or transform on the fly, and the whole pipeline still holds only one line in memory.
For binary files (ZIPs, images, PDFs, large CSV exports) the approach is the same but you read fixed-size chunks instead of lines. Combining pathlib.Path with yield keeps memory flat no matter how large the file is.
from pathlib import Path
from typing import Iterator
def stream_lines(path: str | Path) -> Iterator[str]:
file_path = Path(path)
with file_path.open("r", encoding="utf-8") as f:
for line in f:
yield line
def error_lines(path: str | Path) -> Iterator[str]:
for line in stream_lines(path):
if "ERROR" in line:
yield line.strip()
# --- stand-in for a real file ---
from pathlib import Path
p = Path("app.log")
p.write_text("INFO all good\nERROR disk full\nINFO all good\nERROR timeout\n")
for line in error_lines(p):
print(line)
ERROR disk full ERROR timeout
from pathlib import Path
from typing import Iterator
def stream_file(path: str | Path, chunk_size: int = 4) -> Iterator[bytes]:
file_path = Path(path)
with file_path.open("rb") as f:
while chunk := f.read(chunk_size):
yield chunk
# --- stand-in for a real binary file ---
p = Path("data.bin")
p.write_bytes(b"ABCDEFGHIJ")
for chunk in stream_file(p):
print(chunk)
b'ABCD' b'EFGH' b'IJ'
Finish stream_chunks so it yields one chunk_size-byte chunk at a time without reading the whole object into memory. BytesIO behaves exactly like Path('large-file.bin').open('rb') — swap it in and nothing else changes.
from io import BytesIO
# Stand-in for Path("large-file.bin").open("rb") -- swap it in and nothing else changes.
data = BytesIO(b"ABCDEFGHIJ")
def stream_chunks(file_obj=data, chunk_size=4):
# TODO: yield one chunk at a time without reading everything at once
pass
for chunk in stream_chunks():
print(chunk)Call file_obj.read(chunk_size) in a while loop using the walrus operator: while chunk := file_obj.read(chunk_size):. When the object is exhausted, .read() returns b'' which ends the loop.
from io import BytesIO
# Stand-in for Path("large-file.bin").open("rb") -- swap it in and nothing else changes.
data = BytesIO(b"ABCDEFGHIJ")
def stream_chunks(file_obj=data, chunk_size=4):
while chunk := file_obj.read(chunk_size):
yield chunk
for chunk in stream_chunks():
print(chunk)Chaining generators into a pipeline
Because every generator both consumes an iterable and produces one, you can stack them: each stage does one job — clean, filter, transform — and passes items along lazily. yield from is a shortcut for ‘yield every item of that iterable in turn’, handy for delegating to another source.
def clean(lines):
for line in lines:
yield line.strip()
def to_numbers(lines):
for line in lines:
if line.isdigit():
yield int(line)
raw = ["10\n", "oops\n", "20\n", " 30 \n"]
print(sum(to_numbers(clean(raw))))
60
def chain(*iterables):
for it in iterables:
yield from it
for value in chain([1, 2], (3, 4), "ab"):
print(value)
1 2 3 4 a b
Write a generator running_total(numbers) that yields the total so far after each number. For [1, 2, 3] it should produce 1, 3, 6.
def running_total(numbers):
total = 0
# TODO: add each number to total, then yield the new total
pass
print(list(running_total([1, 2, 3])))Loop the numbers; after total += n, use yield total (not return). The variable total survives across each pause.
def running_total(numbers):
total = 0
for n in numbers:
total += n
yield total
print(list(running_total([1, 2, 3])))Generator functions vs. generator expressions
For a quick one-liner, a generator expression — a comprehension in parentheses — is perfect. Reach for a full generator function when the logic needs more than one expression: multiple steps, state that carries between items, early stops, or yield from. They produce the same kind of lazy, one-shot generator.
numbers = [1, 2, 3, 4, 5]
# Generator expression — parentheses, no function needed:
gen = (n * n for n in numbers)
print(sum(gen))
# The same idea written as a generator function:
def squares(nums):
for n in nums:
yield n * n
print(sum(squares(numbers)))
55 55
Common mistake: Reusing a generator after it’s exhausted
After one full pass (a for loop, list(), or sum()), the generator is empty. Looping it again silently does nothing, which looks like missing data.
Call the generator function again to get a fresh one, or store the values in a list() once if you genuinely need several passes.
Common mistake: Using return instead of yield to send back values
Old habits: a return value inside a generator does not hand that value to the loop — it just stops the generator.
yield each value you want to produce. Use a bare return only when you want to end the generator early.
Common mistake: Calling len() or indexing a generator
A generator produces items on demand, so it has no length and no gen[0] — those raise TypeError.
If you truly need length or random access, convert with list(gen) first (you lose the lazy memory savings). Otherwise just iterate it.
What does calling a generator function, like count_up_to(3), actually do?
Calling it builds a generator; the body only runs as you pull values out with a loop, next(), or list().
You loop a generator fully, then loop it again. What happens the second time?
Generators are one-shot. To go through the values again, call the generator function again for a fresh one.
When is a generator the better choice over a list?
Generators shine for streaming large or lazy data with low memory. For random access or multiple passes, a list is the right tool.
Mini exercise (medium)
Write a generator first_words(lines) that yields the first word of each non-blank line. Strip each line first, and skip lines that are empty or only spaces.
Do it here. Finish the code below and press Run to try your answer right in the browser — no setup needed.
def first_words(lines):
# TODO: for each non-blank line, yield its first word (stripped)
pass
print(list(first_words(["hello world", " ", "foo bar baz"])))Loop the lines; set line = line.strip(), continue if it's empty, otherwise yield line.split()[0].
def first_words(lines):
for line in lines:
line = line.strip()
if not line:
continue
yield line.split()[0]
print(list(first_words(["hello world", " ", "foo bar baz"])))