课程 · 12 · 07 / 12
LESSON 07 · ADVANCED · 55 MIN · ◆ 2 INSTRUMENTS
Multiprocessing: True Parallelism in Python
Bypass the GIL with multiprocessing. Use process pools, manage shared state safely, and understand inter-process communication.
TIPLearning Objectives: After this lesson, you'll understand how to bypass the GIL with multiprocessing, use process pools efficiently, manage shared state safely, and implement inter-process communication patterns.
Why Multiprocessing?
Multiprocessing creates separate Python processes, each with its own GIL, enabling true parallelism.
FIG. 02Flow Diagram
INTERACTIVE
LOADING INSTRUMENT
Fig. 02Interactive flow diagrams, timelines, and process visualizations
Multiprocessing vs Threading
FIG. 04Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 04Interactive Python code execution environment
Basic Process Creation
FIG. 06Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 06Interactive Python code execution environment
Process Pools
Process pools manage a fixed number of worker processes for efficient task distribution.
FIG. 08Flow Diagram
INTERACTIVE
LOADING INSTRUMENT
Fig. 08Interactive flow diagrams, timelines, and process visualizations
Using Pool
FIG. 10Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 10Interactive Python code execution environment
Pool Methods
FIG. 12Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 12Interactive Python code execution environment
Shared State and Communication
Processes don't share memory by default. Here's how to share data.
FIG. 14Flow Diagram
INTERACTIVE
LOADING INSTRUMENT
Fig. 14Interactive flow diagrams, timelines, and process visualizations
Shared Memory Values
FIG. 16Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 16Interactive Python code execution environment
Manager for Complex Shared Objects
FIG. 18Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 18Interactive Python code execution environment
Queues for Communication
FIG. 20Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 20Interactive Python code execution environment
Pipes for Two-Way Communication
FIG. 22Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 22Interactive Python code execution environment
Practical Patterns
Map-Reduce Pattern
FIG. 24Flow Diagram
INTERACTIVE
LOADING INSTRUMENT
Fig. 24Interactive flow diagrams, timelines, and process visualizations
FIG. 26Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 26Interactive Python code execution environment
Parallel Pipeline
FIG. 28Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 28Interactive Python code execution environment
Worker Pool with Callbacks
FIG. 30Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 30Interactive Python code execution environment
Best Practices
FIG. 32Python Code Executor
INTERACTIVE
LOADING INSTRUMENT
Fig. 32Interactive Python code execution environment
Key Takeaways
| Concept | Description |
|---|---|
| Process | Separate Python interpreter, own GIL |
| Pool | Reusable worker processes |
| ProcessPoolExecutor | Modern, clean pool interface |
| Value/Array | Shared memory primitives |
| Manager | Shared complex objects (dict, list) |
| Queue | Thread/process-safe communication |
| Pipe | Two-way process communication |
Multiprocessing vs Alternatives
| Scenario | Best Choice |
|---|---|
| CPU-heavy calculation | Multiprocessing |
| Network I/O | Threading or asyncio |
| File I/O | Threading |
| Many small tasks | Pool with chunksize |
| Large shared data | SharedMemory (Python 3.8+) |
| NumPy operations | NumPy (already parallel) |
Next Steps
In the next lesson, we'll explore Async/Await Fundamentals—understand event loops, write coroutines, use asyncio for concurrent I/O, and build async patterns for production code.
Ready for non-blocking Python? Async/await awaits!
Further Reading
Official Docs
- Python —
multiprocessingmodule —Process,Pool,Queue,Pipe,Manager,shared_memory. - Python —
concurrent.futures.ProcessPoolExecutor— the modern high-level API. Use this before rawProcess. - Python —
multiprocessing.shared_memory— Python 3.8+. Share NumPy arrays without pickling overhead.
Tutorials
- Real Python — Speed Up Your Python Program with Concurrency — the canonical comparison of threading / multiprocessing / asyncio.
- Real Python —
multiprocessing.Pool— the pool patterns deep-dive.
Modern Parallel Python
joblib— the parallelism library used by scikit-learn. Cleaner API than raw multiprocessing for embarrassingly-parallel work.ray— distributed Python. Scales from one machine to a cluster with@ray.remote.dask— parallel computing for analytics; familiar pandas/numpy APIs.mpire— modern multiprocessing wrapper with progress bars and worker reuse.
Common Pitfalls
- The "fork" vs "spawn" vs "forkserver" debate — macOS defaults to
spawnsince 3.8 for good reasons. Know which you're using. - Pickling errors — only picklable objects can cross process boundaries. Lambdas and local functions can't.
Books
- Book: Fluent Python (2nd ed.) — Chapter 20 ("Concurrent Executors").
- Book: High Performance Python — Gorelick & Ozsvald (2nd ed., 2020). Covers when multiprocessing actually helps vs. just adding overhead.