The most mind-bending thing about Python generators is that they are both iterators and state machines, and you don’t have to explicitly manage the state yourself.
Let’s see one in action. Imagine you want to process a massive log file, but you don’t want to load the whole thing into memory. A generator is perfect for this:
def read_log_lines(filepath):
with open(filepath, 'r') as f:
for line in f:
yield line.strip()
log_file = 'massive.log'
log_reader = read_log_lines(log_file)
# You can iterate over it like a list, but it's memory efficient
for log_line in log_reader:
if "ERROR" in log_line:
print(f"Found error: {log_line}")
# You can also call next() on it to get items one by one
first_line = next(log_reader)
print(f"The next line is: {first_line}")
This read_log_lines function is a generator. When you call it, it doesn’t execute the code inside immediately. Instead, it returns a generator object. This object is an iterator, meaning it has a __next__ method that, when called, resumes the generator’s execution from where it last yielded. The yield keyword is the magic. It pauses the function’s execution, saves its entire local state (variables, instruction pointer), and returns a value. When next() is called again, execution resumes right after the yield statement.
The problem generators solve is elegantly handling sequences of data that are too large to fit into memory, or sequences that are computationally expensive to generate all at once. Think infinite sequences, or streams of data from a network socket. They provide a clean, Pythonic way to "pull" data as needed, rather than "pushing" it all upfront.
Internally, when you yield a value, Python creates a "frame" object that captures the complete state of the generator function at that point. This frame includes all local variables, the current line of execution, and the generator’s internal state (whether it’s paused, running, or finished). When next() is called, Python retrieves this frame and resumes execution. If the generator function finishes without hitting another yield, it raises StopIteration, signaling that the iteration is complete.
Coroutines, which are closely related, build upon this by allowing functions to receive data as well as send it. The yield keyword can be used on the right-hand side of an assignment to receive a value from the caller. This enables a two-way communication channel.
def echo_coroutine():
print("Coroutine started. Waiting for input...")
while True:
received_data = yield # Yields None initially, then receives data
if received_data:
print(f"Coroutine received: {received_data}")
else:
print("Coroutine received no data. Exiting.")
break
coro = echo_coroutine()
next(coro) # Prime the coroutine: starts execution, reaches 'yield', pauses.
coro.send("Hello, world!") # Sends data to the coroutine, resumes it.
coro.send("Another message.")
coro.send(None) # Sends None, which will cause the coroutine to break.
The send() method is key here. Calling coro.send("Hello, world!") does two things: it sends the string "Hello, world!" into the coroutine, making it the value of received_data in the next iteration, and it resumes the coroutine’s execution. The first next(coro) call is crucial to "prime" the coroutine, getting it to the yield statement so it’s ready to receive data. Without it, send() would raise a TypeError.
The most subtle aspect of generators and coroutines is how yield works with send() and throw(). While send(value) injects value into the coroutine, throw(exception) can inject an exception into the coroutine at the point where it’s paused. This allows for sophisticated error handling and control flow within the generator itself. For example, you could have a try...except block around a yield statement inside the generator to catch exceptions sent from the caller.
The next frontier you’ll encounter is using asyncio and await, which are built on coroutine principles but introduce cooperative multitasking for I/O-bound operations.