How for Loops Really Work: The Iterator Protocol
What Python has been doing behind every loop
In this lesson
You have written hundreds of for loops without needing to know how they work. This lesson opens the lid. Underneath is a small, simple agreement called the iterator protocol, and once you can see it, generators, itertools, and half of Python's memory-efficiency story stop being magic.
Explain it like I’m 5
What a for loop actually does
When you write for item in things:, Python does three things you never see. It calls iter(things) to get an iterator. Then it calls next() on that iterator over and over. When there is nothing left, next() raises StopIteration, and the loop treats that as ‘stop, quietly’.
That is the entire protocol. Anything that can answer those two calls can be used in a for loop, which is why the same syntax works on lists, files, dictionaries, and things you write yourself.
things = ["a", "b"]
# What the for loop does for you:
it = iter(things)
print(next(it))
print(next(it))
try:
next(it)
except StopIteration:
print("that was the last one")
a b that was the last one
An iterator is used up after one pass
This is the part that surprises people, and it follows directly from the protocol. An iterator remembers its position. Once it has handed out the last item, it is exhausted, and looping over it again gives you nothing at all — not an error, just an empty loop.
A list is different: it is iterable but not itself an iterator, so iter() gives you a fresh iterator each time. That is why you can loop over a list as many times as you like.
numbers = [1, 2, 3]
it = iter(numbers)
print("first pass: ", list(it))
print("second pass:", list(it)) # already exhausted
print("the list itself:", list(numbers))
print("again: ", list(numbers))
first pass: [1, 2, 3] second pass: [] the list itself: [1, 2, 3] again: [1, 2, 3]
Writing your own
To make your own class work in a for loop, give it two methods. __iter__ returns the iterator (usually self), and __next__ returns the next value or raises StopIteration when it is done.
These are the dunder methods from Unit 4, doing the same job as always: letting your object join in with normal Python syntax.
class Countdown:
"""Counts down from a number to 1."""
def __init__(self, start):
self.current = start
def __iter__(self):
return self # I am my own iterator
def __next__(self):
if self.current <= 0:
raise StopIteration # nothing left: stop the loop
self.current -= 1
return self.current + 1
for number in Countdown(3):
print(number)
3 2 1
Write an iterable that returns every third item from a list. Fill in __next__: raise StopIteration once self.index has run past the end, otherwise return the item there and move the index on by three.
class EveryThird:
def __init__(self, items):
self.items = items
self.index = 0
def __iter__(self):
return self
def __next__(self):
pass # TODO: stop past the end, else return the item and step by 3
for value in EveryThird(["a", "b", "c", "d", "e", "f", "g"]):
print(value)Start with if self.index >= len(self.items): raise StopIteration. Then grab self.items[self.index] into a variable, add 3 to self.index, and return the variable. Save the value before you move the index.
class EveryThird:
def __init__(self, items):
self.items = items
self.index = 0
def __iter__(self):
return self
def __next__(self):
if self.index >= len(self.items):
raise StopIteration
value = self.items[self.index]
self.index += 3
return value
for value in EveryThird(["a", "b", "c", "d", "e", "f", "g"]):
print(value)When a generator is the better answer
Now the honest part. Almost every custom iterator you would write is shorter as a generator, the yield from Unit 5. The protocol is what generators are built on, and knowing it is what lets you read tracebacks and understand itertools. But in day-to-day code, reach for yield.
The real payoff of both is memory. An iterator holds one item at a time, so streaming a huge file costs the same as streaming a tiny one.
# The Countdown class above, as a generator:
def countdown(start):
while start > 0:
yield start
start -= 1
print(list(countdown(3)))
# Ten million numbers, one at a time. Memory stays flat.
def squares(limit):
for n in range(limit):
yield n * n
big = squares(10_000_000)
print(next(big), next(big), next(big))
[3, 2, 1] 0 1 4
Common mistake: Looping over the same iterator twice
It looks like a collection, and collections can be looped repeatedly.
An iterator is consumed as you go. Capture the results with list(...) if you need more than one pass, or create a fresh iterator.
Common mistake: Forgetting to raise StopIteration
__next__ works for the first few calls, so the loop appears to run.
Without it the loop never ends. Make the stop condition the first thing __next__ checks.
Common mistake: Moving the index before saving the value
Updating state first feels tidy.
Read the current item into a variable, then advance, then return the variable. Otherwise you skip the first item and return the wrong one.
Common mistake: Writing a class where a generator would do
The protocol is what you have just learned, so it is the tool in hand.
Use yield unless you genuinely need an object with other methods and state. The generator is shorter and harder to get wrong.
What does a for loop call first on the object you give it?
iter() comes first and produces the iterator; next() is then called repeatedly on that.
What signals that iteration is finished?
StopIteration is the agreed signal, and the for loop catches it for you.
Why does looping a second time over the same iterator produce nothing?
An iterator tracks where it is. Once exhausted it keeps reporting that it is finished.
Mini exercise (medium)
Write a Repeat iterable class taking a value and a count, so list(Repeat("hi", 3)) gives ['hi', 'hi', 'hi']. Then write the same thing as a generator function called repeat_value and confirm both produce the same list.
Give it a shot. Complete the code and press Run; there’s nothing to download or configure.
class Repeat:
def __init__(self, value, count):
self.value = value
self.remaining = count
def __iter__(self):
return self
def __next__(self):
pass # TODO: stop at zero, else count down and return the value
def repeat_value(value, count):
pass # TODO: the same thing as a generator
print(list(Repeat("hi", 3)))
print(list(repeat_value("hi", 3)))
print(list(Repeat("hi", 3)) == list(repeat_value("hi", 3)))The class needs __init__ to store the value and a remaining count, __iter__ returning self, and __next__ raising StopIteration when the count hits zero. The generator is a for loop with a single yield.
class Repeat:
def __init__(self, value, count):
self.value = value
self.remaining = count
def __iter__(self):
return self
def __next__(self):
if self.remaining <= 0:
raise StopIteration
self.remaining -= 1
return self.value
def repeat_value(value, count):
for _ in range(count):
yield value
print(list(Repeat("hi", 3)))
print(list(repeat_value("hi", 3)))
print(list(Repeat("hi", 3)) == list(repeat_value("hi", 3)))