Learning Objectives: After this lesson, you'll understand the iterator protocol, build generators with yield, use generator expressions efficiently, and see how generators form the foundation for coroutines and async programming.
Understanding Lazy vs Eager Evaluation
Before diving into code, let's visualize the fundamental difference between eager (list) and lazy (generator) evaluation:
With lists, all values are computed and stored upfront. With generators, values are computed one at a time as needed, keeping memory usage constant.
The Iterator Protocol
Python's iteration is built on a simple protocol: __iter__ and __next__.
Understanding Iteration
Building a Custom Iterator
Iterable vs Iterator
Generators: Iterators Made Simple
Generators provide a concise way to create iterators using yield. Let's visualize how a generator suspends and resumes:
Basic Generators
Step Through Generator Execution
Use the algorithm stepper below to trace through generator execution step by step:
Memory Comparison: Generators vs Lists
Generator Expressions
Generator expressions provide a concise syntax for simple generators.
Advanced Generator Features
yield from - Delegating to Sub-generators
Generator Send and Throw
Generators can also receive values, enabling two-way communication:
Generator Close and Cleanup
Practical Generator Patterns
Data Pipeline Processing
The key insight: data flows on demand. The consumer pulls values through the pipeline, and each stage only processes one item at a time.
Infinite Sequences
Memory-Efficient File Processing
Generators and itertools
Practice Exercises
Exercise 1: Sliding Window Generator
Exercise 2: Tree Traversal Generator
Key Takeaways
| Concept | Description |
|---|---|
| Iterator Protocol | __iter__ returns iterator, __next__ returns next value |
| Generator Function | Function with yield, returns generator object |
| Generator Expression | (expr for x in iterable) - lazy list comprehension |
yield from | Delegate to sub-generator, simplifies nested iteration |
send() | Pass values INTO generator (coroutine pattern) |
| Lazy Evaluation | Values computed on-demand, memory efficient |
When to Use Generators
- Large datasets - Process without loading everything into memory
- Infinite sequences - Fibonacci, primes, streaming data
- Data pipelines - Chain transformations lazily
- File processing - Line-by-line without loading entire file
- API pagination - Fetch pages on-demand
Next Steps
In the next lesson, we'll explore Context Managers and Descriptors—master the with statement, build custom context managers, and understand how descriptors power Python's attribute access.
Ready to manage resources like a pro? Context managers await!