Advanced Python: What's Left, and When You'll Need It

Six features past this course, each with the problem that earns it

Advanced 17 min

In this lesson

“Advanced Python” is usually presented as a locked room. It is not: it is a set of tools with unusually narrow jobs, and the reason you have not met them is that you have not yet had the problems they solve.

This lesson gives you one small example of each, clearly labeled as a preview. You are not expected to master any of it today. You are expected to recognize the smell of the problem later, and know what to look up.

Explain it like I’m 5

These are the specialist tools at the back of the toolbox. You do not carry them around. You go and get one when a specific job turns up.

You already carry most of the advanced toolbox

Start with the confidence beat, because it is deserved. Almost everything that gets called advanced Python, you have done:

That is most of any “advanced Python” syllabus you will find. What follows is the genuine remainder: six things this course has not taught, in roughly the order you are likely to need them, which is also roughly the order of how rarely you will.

__slots__: when a million small objects will not fit

Every ordinary Python object carries a __dict__ — a dictionary of its attributes — which is what lets you add an attribute to an instance whenever you like. It costs memory per object, and you do not notice until there are a lot of objects.

__slots__ declares the attribute names up front, and the class stops carrying that dictionary.

Example 
class Point:
    """A point that may only ever have an x and a y."""
    __slots__ = ("x", "y")

    def __init__(self, x, y):
        self.x = x
        self.y = y

point = Point(1, 2)
print("x, y:", point.x, point.y)

try:
    point.z = 3                      # not in __slots__
except AttributeError:
    print("blocked: Point has no slot named z")

print("has a __dict__:", hasattr(point, "__dict__"))
Output 
x, y: 1 2
blocked: Point has no slot named z
has a __dict__: False
Smaller objects, and a class that refuses attributes you did not declare.

Descriptors: you have already used one

Whenever you have written @property, you used a descriptor. A descriptor is any object that defines __get__ or __set__ and is then used as a class attribute: Python routes attribute access through it. property, classmethod, and even plain methods are all descriptors underneath.

You write your own when the same managed-attribute logic is needed on several attributes and a property per attribute would mean copying it.

Example 
class Positive:
    """A managed attribute that refuses a negative number."""

    def __set_name__(self, owner, name):
        self.storage = "_" + name

    def __get__(self, instance, owner=None):
        return getattr(instance, self.storage)

    def __set__(self, instance, value):
        if value < 0:
            raise ValueError(f"{self.storage[1:]} cannot be negative: {value}")
        setattr(instance, self.storage, value)

class Order:
    quantity = Positive()            # one descriptor, guarding every assignment

    def __init__(self, quantity):
        self.quantity = quantity

order = Order(3)
print("quantity:", order.quantity)

order.quantity = 10
print("updated: ", order.quantity)

try:
    order.quantity = -1
except ValueError as error:
    print("refused: ", error)
Output 
quantity: 3
updated:  10
refused:  quantity cannot be negative: -1
The check runs on every assignment, including the one inside __init__.

Protocols: type-checking duck typing

Python has always cared about what an object can do rather than what it is. If it has a close() method, you can close it, whatever its class. That is duck typing, and it is why so little Python needs inheritance.

The problem was that type hints could not express it: you could write def shut(thing: Connection), which is a lie if anything closable will do. typing.Protocol fixes that — it describes a shape rather than an ancestry.

Example 
from typing import Protocol, runtime_checkable

@runtime_checkable
class Closable(Protocol):
    """Anything with a close() method, whatever its class."""
    def close(self) -> None: ...

class Connection:
    def close(self):
        return "closed"

class Report:
    def render(self):
        return "<html>"

for thing in (Connection(), Report()):
    print(f"{type(thing).__name__:11s} closable: {isinstance(thing, Closable)}")
Output 
Connection  closable: True
Report      closable: False
Connection never mentions Closable, and still satisfies it.

Metaclasses: the one you will probably never need

A class is itself an object, and something has to create it. That something is a metaclass, and by default it is type. Writing your own means running code at the moment a class is defined, rather than when it is used.

The classic legitimate use is a registry: every subclass should announce itself somewhere without the author having to remember.

Example 
class Registry(type):
    """A metaclass runs when a CLASS is created, not when one is used."""
    exporters = {}

    def __init__(cls, name, bases, namespace):
        super().__init__(name, bases, namespace)
        if bases:                    # skip the base class itself
            Registry.exporters[name.removesuffix("Exporter").lower()] = cls

class Exporter(metaclass=Registry):
    pass

class CsvExporter(Exporter):
    pass

class JsonExporter(Exporter):
    pass

print("registered:", sorted(Registry.exporters))
print("csv is:    ", Registry.exporters["csv"].__name__)
Output 
registered: ['csv', 'json']
csv is:     CsvExporter
Nobody called register(). Defining the class was enough.
Example 
class Exporter:
    """The same registry, with no metaclass in sight."""
    exporters = {}

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        Exporter.exporters[cls.__name__.removesuffix("Exporter").lower()] = cls

class CsvExporter(Exporter):
    pass

class JsonExporter(Exporter):
    pass

print("registered:", sorted(Exporter.exporters))
Output 
registered: ['csv', 'json']
Same result, an ordinary class, and one method instead of a metaclass.

Profiling: measure first, then optimize

This is the one on the list you will actually use, and the rule is short: never optimize before measuring. Intuition about which line is slow is wrong often enough to be worthless.

Before reaching for a profiler, though, there is a cheaper insight. Most real slowness is not about how fast each operation runs; it is about how many operations you do. Counting them is something you can do in your head, or here.

Example 
NAMES = [f"user{n:04d}" for n in range(2000)]
KNOWN = set(NAMES)
LOOKUPS = ["user0000", "user1000", "user1999", "nobody"]

def comparisons_to_find(names, wanted):
    """How many names a plain list scan has to look at."""
    for position, name in enumerate(names, start=1):
        if name == wanted:
            return position
    return len(names)

print("comparisons to find a name in a LIST of 2000:")
for wanted in LOOKUPS:
    print(f"  {wanted:9s} {comparisons_to_find(NAMES, wanted):5d}")

print("comparisons to find the same name in a SET:")
for wanted in LOOKUPS:
    print(f"  {wanted:9s} {1:5d}    found: {wanted in KNOWN}")
Output 
comparisons to find a name in a LIST of 2000:
  user0000      1
  user1000   1001
  user1999   2000
  nobody     2000
comparisons to find the same name in a SET:
  user0000      1    found: True
  user1000      1    found: True
  user1999      1    found: True
  nobody        1    found: False
The same question, asked of two collections.
Example 
$ python -m cProfile -s cumtime slow.py
5000
         10009 function calls in 0.233 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    0.233    0.233 {built-in method builtins.exec}
        1    0.003    0.003    0.233    0.233 slow.py:1(<module>)
        1    0.000    0.000    0.230    0.230 slow.py:19(main)
        1    0.000    0.000    0.229    0.229 slow.py:11(check_all)
        1    0.000    0.000    0.229    0.229 {built-in method builtins.sum}
     5001    0.001    0.000    0.228    0.000 slow.py:12(<genexpr>)
     5000    0.228    0.000    0.228    0.000 slow.py:6(is_known)
        1    0.001    0.001    0.001    0.001 slow.py:15(load)
cProfile on a script that felt slow, sorted by cumulative time (trimmed).
Example 
$ python -m timeit -s "NAMES=[f'user{n:05d}' for n in range(20000)]" "'user19999' in NAMES"
5000 loops, best of 5: 85.2 usec per loop

$ python -m timeit -s "KNOWN={f'user{n:05d}' for n in range(20000)}" "'user19999' in KNOWN"
50000000 loops, best of 5: 7.98 nsec per loop
timeit for the smallest possible comparison: one lookup, two collections.

C extensions, and why you probably will not write one

When a profiler proves that one small function is the whole bottleneck and no algorithm change helps, you can drop out of Python for that function. The usual routes are Cython (Python-like code compiled to C), ctypes or cffi (call an existing C library), the raw CPython C API, and increasingly Rust through PyO3.

Here is the honest position: for numeric work, the answer is almost always NumPy or pandas, which are already C underneath. Somebody already wrote the fast version, tested it on more hardware than you have, and gave it a Python interface. Learning to reach for those is the same win with none of the build system.

What is worth knowing today is just the shape of the escape hatch: Python can call into compiled code, that is how its whole scientific stack is built, and the door is there if you ever genuinely need it.

Fix this preview in place. For each of the six features, the table below already names the problem that earns it — fill in the simpler thing you already know and the unit that taught it, so the feature has something to be compared against.

# The order here is the order you are likely to need them.
ORDER = ["__slots__", "descriptor", "Protocol", "metaclass", "cProfile", "C extension"]

EARNS_IT = {
    "__slots__": "millions of tiny objects will not fit in memory",
    "descriptor": "every assignment to one attribute must be checked",
    "Protocol": "type-check duck typing without a shared base class",
    "metaclass": "every subclass must register itself as it is defined",
    "cProfile": "it is slow and guessing which line has not worked",
    "C extension": "one function is provably the whole bottleneck",
}

# TODO: the simpler thing you already know, and the unit that taught it.
#       Choose from: a plain class / property / duck typing /
#       __init_subclass__, or a class decorator / timing one block by hand /
#       numpy, which is already C
ALREADY_HAVE = {
    "__slots__": ("?", 0),
    "descriptor": ("?", 0),
    "Protocol": ("?", 0),
    "metaclass": ("?", 0),
    "cProfile": ("?", 0),
    "C extension": ("?", 0),
}

for feature in ORDER:
    simpler, unit = ALREADY_HAVE[feature]
    print(f"{feature:12s} unit {unit:2d}  you already have: {simpler}")
    print(f"{'':12s} reach past it when: {EARNS_IT[feature]}")

Common mistake: Learning an advanced feature before having a problem for it

Why it happens:

The features have a reputation, so knowing them feels like a level-up.

How to fix it:

Read this lesson once for recognition, then leave it. Each of these tools is easy to learn on the day you need it and almost impossible to retain on a day you do not.

Common mistake: Writing a metaclass where a decorator or __init_subclass__ would do

Why it happens:

A metaclass is the answer most search results give for “run code when a class is defined”, often from before __init_subclass__ existed.

How to fix it:

Try __init_subclass__ first, then a class decorator. If neither can express it, you have one of the rare real cases — and you will be able to say exactly why in one sentence.

Common mistake: Optimizing before profiling

Why it happens:

One line looks expensive, and rewriting it feels like progress.

How to fix it:

Run cProfile first, every time. It takes ten seconds and it is regularly a different function than the one you suspected — often one you did not know was being called 5,000 times.

Common mistake: Adding __slots__ to every class

Why it happens:

It reads as a free optimization and a tidiness win.

How to fix it:

It is a real constraint: no new attributes, awkward inheritance, and surprises with libraries that expect to attach data to your objects. Use it where a measurement showed memory pressure from many instances.

Common mistake: Assuming advanced features make code better

Why it happens:

They are what distinguishes expert-looking code in examples.

How to fix it:

They distinguish expert code because experts reach for them rarely and precisely. The most senior-looking Python you will ever read is mostly plain functions with good names.

Which unit taught you decorators?

What should you do before optimizing a slow program?

When is a metaclass the right tool?

What does __slots__ actually cost you?

Mini exercise (hard)

Write the registry from this lesson the way you actually should: with __init_subclass__ and no metaclass. Every Exporter subclass must add itself to Exporter.registry under its own name minus the Exporter suffix, lower-cased. Then export(format_name, rows) looks the class up, uses it, and reports what is available when the name is unknown.

Give it a shot. Complete the code and press Run; there’s nothing to download or configure.

class Exporter:
    """Base class. Every subclass should register itself automatically."""
    registry = {}

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        # TODO: register cls under its name minus "Exporter", lower-cased

    def render(self, rows):
        raise NotImplementedError

class CsvExporter(Exporter):
    def render(self, rows):
        return "\n".join(",".join(row) for row in rows)

class TsvExporter(Exporter):
    def render(self, rows):
        return "\n".join("\t".join(row) for row in rows)

def export(format_name, rows):
    """Render rows with the named exporter, or say what is available."""
    return "?"     # TODO: look format_name up in the registry and use it
                   # TODO: unknown name -> f"no {format_name} exporter; have: ..."

ROWS = [("name", "city"), ("ada", "oxford")]
print("formats:", sorted(Exporter.registry))
print(export("csv", ROWS))
print(export("tsv", ROWS).replace("\t", " | "))
print(export("pdf", ROWS))

What to learn next

You have met the genuine remainder of advanced Python: __slots__ for when a million objects will not fit, descriptors for validating every assignment to an attribute, protocols for type-checking duck typing, metaclasses and the __init_subclass__ that has replaced most of them, profiling before optimizing, and the C-extension escape hatch you will probably never need. You also wrote the registry the modern way, and watched a class defined afterwards register itself.

Every one of those is something you look up on the day it becomes relevant, which makes finding things out the real skill. That is next: Reading Documentation Without Getting Lost turns it into a method.