Module 03: Iterators, Laziness, and Streaming Dataflow

Page Maps

graph LR
  family["Python Programming"]
  program["Python Functional Programming"]
  section["Iterators Laziness Streaming Dataflow"]
  page["Module 03: Iterators, Laziness, and Streaming Dataflow"]
  capstone["Capstone evidence"]

  family --> program --> section --> page
  page -.applies in.-> capstone

flowchart LR
  orient["Orient on the page map"] --> read["Read the main claim and examples"]
  read --> inspect["Inspect the related code, proof, or capstone surface"]
  inspect --> verify["Run or review the verification path"]
  verify --> apply["Apply the idea back to the module and capstone"]

Read the first diagram as a placement map: this page sits between the course promise, the lesson pages listed below, and the capstone surfaces that pressure-test the module. Read the second diagram as the study route for this page, so the diagrams point you toward the Lesson map, Exercises, and Closing criteria instead of acting like decoration.

Keep These Pages Open

Use these support surfaces while reading so laziness stays tied to execution timing, materialization, and proof instead of becoming stream vocabulary on its own:

• First-Contact Map for the full foundation route that ends here
• Start Here for the paced route through Modules 01 to 03
• Proof Matrix for the smallest honest evidence route
• Capstone Map for the streaming and tree-fold surfaces in FuncPipe

Carry this question into the module:

When does work happen, where does materialization occur, and how can I make that timing visible enough to review?

This module makes streaming a first-class part of the course architecture. You move from pure transforms over finite collections to deliberate control over when work happens, how much memory is used, and where materialization becomes a conscious choice.

Start with FuncPipe Delta. It identifies the exact Module 02 behavior that must survive, the new streaming surfaces, the proof route, and the limits deliberately handed to later modules.

Learning outcomes

• how iterators and generators model on-demand dataflow in Python
• how itertools and custom iterators support reusable streaming stages
• how to reason about chunking, fan-in, fan-out, and bounded traversal
• how to add observability to lazy pipelines without destroying laziness

Lesson map

• Iterator Protocol and Generators
• Generators vs Comprehensions
• itertools Composition
• Chunking and Windowing
• Infinite Sequences Safely
• Reusable Pipeline Stages
• Pipeline Stage Review and Reuse
• Fan-In and Fan-Out
• Time-Aware Streaming
• Custom Iterators
• Iterator Lifecycle and Cleanup
• Streaming Observability
• FuncPipe Delta
• Exercises
• Exercise Answers
• Refactoring Guide

Exercises

Use Module 03 Exercises for six cumulative FuncPipe reviews: demand tracing, chunk metadata, document-source scheduling, bounded fan-out, stage ownership, and safe observation. Each exercise requires value evidence and an execution-policy argument. Compare with Exercise Answers only after writing your own prediction.

Capstone checkpoints

• Trace stream_rag_chunks through cleaning, chunking, embedding, encounter-order deduplication, and the final unique-output fence.
• Compare gen_bounded_chunks with stream_rag_chunks; state what each limit actually bounds.
• Compare the complete stream with full_rag_api_docs under canonical and out-of-order input.
• Trace stream_rag_sources through sequential and round-robin scheduling; state which source factories open and which documents are pulled for one output.
• Run make PROGRAM=python-programming/python-functional-programming capstone-source-scheduling-proof and explain all six multi-source assertions.
• Run make PROGRAM=python-programming/python-functional-programming capstone-streaming-rag-proof and explain both the duplicate-demand and multicast assertions.

Before moving on

You should be able to explain why laziness changes error handling, resource management, and review strategy before the course introduces typed failures and resilience patterns. Use Refactoring Guide and compare against capstone/module-reference-states/module-03 before moving forward.

Closing criteria

• You can explain when work happens in a pipeline instead of only what data moves through it.
• You can review a streaming helper and spot hidden materialization, cleanup leaks, or ordering surprises.
• You can justify the boundary between reusable lazy stages and explicit materialization.
• You can defend sequential or round-robin document scheduling from an application ordering requirement.
• You can explain why a final unique-output limit does not necessarily bound raw upstream work one-for-one.

Directory glossary

Use Glossary when you want the recurring language in this module kept stable while you move between lessons, exercises, and capstone checkpoints.