Generators and Iterators: Lazy Evaluation Mastery

TIP

Learning Objectives: After this lesson, you'll understand the iterator protocol, build generators with yield, use generator expressions efficiently, and see how generators form the foundation for coroutines and async programming.

Understanding Lazy vs Eager Evaluation

Before diving into code, let's visualize the fundamental difference between eager (list) and lazy (generator) evaluation:

With lists, all values are computed and stored upfront. With generators, values are computed one at a time as needed, keeping memory usage constant.

The Iterator Protocol

Python's iteration is built on a simple protocol: __iter__ and __next__.

Understanding Iteration

Building a Custom Iterator

Iterable vs Iterator

Use the memory visualizer above to explore how iterables and iterators differ in memory. Iterables can create multiple iterators; iterators maintain state and are single-use.

Generators: Iterators Made Simple

Generators provide a concise way to create iterators using yield. Let's visualize how a generator suspends and resumes:

Basic Generators

Step Through Generator Execution

Use the algorithm stepper below to trace through generator execution step by step:

How yield Works

Memory Comparison: Generators vs Lists

Generator Expressions

Generator expressions provide a concise syntax for simple generators.

Advanced Generator Features

yield from - Delegating to Sub-generators

Generator Send and Throw

Generators can also receive values, enabling two-way communication:

Generator Close and Cleanup

Practical Generator Patterns

Data Pipeline Processing

The key insight: data flows on demand. The consumer pulls values through the pipeline, and each stage only processes one item at a time.

Infinite Sequences

Memory-Efficient File Processing

Generators and itertools

Practice Exercises

Exercise 1: Sliding Window Generator

Exercise 2: Tree Traversal Generator

Key Takeaways

When to Use Generators

• Large datasets - Process without loading everything into memory
• Infinite sequences - Fibonacci, primes, streaming data
• Data pipelines - Chain transformations lazily
• File processing - Line-by-line without loading entire file
• API pagination - Fetch pages on-demand

Next Steps

In the next lesson, we'll explore Context Managers and Descriptors—master the with statement, build custom context managers, and understand how descriptors power Python's attribute access.

Ready to manage resources like a pro? Context managers await!

Further Reading

Visualize It

• Python Tutor — paste a generator function and step through next() calls. Watch local state suspend and resume — the most underrated Python feature in plain sight.

Official Docs

• Python — yield Expression — the formal language reference.
• Python — itertools module — every iterator combinator you'll ever need.
• Python — more-itertools — what itertools would be if it had 200 functions instead of 30.
• PEP 380 — yield from — sub-generator delegation.

Tutorials

• David Beazley — Generator Tricks for Systems Programmers — the legendary slide deck. Read it; your sense of what generators can do will multiply.
• Real Python — How to Use Generators and yield — the canonical walkthrough.
• Trey Hunner — Iterating in Python — iterator protocol explained gently.

Modern Use Cases

• Async generators (async def + yield) — used by FastAPI/Starlette streaming responses, OpenAI's streaming API client, etc. PEP 525.
• PEP 530 — Asynchronous Comprehensions — [x async for x in stream].

Books

• Book: Fluent Python (2nd ed.) — Chapter 17 ("Iterators, Generators, and Classic Coroutines"). The definitive treatment.
• Book: Effective Python — Items 30–34 cover generators in depth.