Iterators & Generators

What you'll learn

How does a for loop actually work? And how can you process a million-line file, or even an infinite sequence, without running out of memory? The answers are iterators and generators — one of Python's most elegant and practical features.

By the end you'll be able to:

• Explain the iterator protocol behind every for loop
• Write generators with yield
• Produce lazy and even infinite sequences
• Build memory-efficient generator pipelines with itertools

The iterator protocol

An iterable is anything you can loop over (list, string, dict, file). Calling iter() on it gives an iterator — an object with a __next__ method that returns the next value, and raises StopIteration when exhausted. A for loop is just iter() + repeated next() with the stop handled for you:

nums = [10, 20, 30]
it = iter(nums)          # get an iterator
print(next(it))          # 10
print(next(it))          # 20
print(next(it))          # 30
# next(it)               # StopIteration — the loop's stop signal

Generators & yield

Writing a full iterator class is tedious. A generator is the easy way: an ordinary function that uses yield instead of return. Each yield pauses the function and hands back a value; the next request resumes right where it left off, with all local state intact:

def countdown(n):
    while n > 0:
        yield n          # pause here, hand back a value, resume next time
        n -= 1

for x in countdown(3):
    print(x)

print(list(countdown(5)))    # materialize into a list

Lazy & infinite sequences

Because values are produced on demand, a generator can represent an infinite sequence — you just take what you need. And a generator expression (a comprehension in parentheses) gives you the same laziness with comprehension syntax:

def naturals():
    n = 1
    while True:          # infinite! but lazy — values made on demand
        yield n
        n += 1

gen = naturals()
print([next(gen) for _ in range(5)])   # take just the first 5

# Generator expression: like a list comp, but lazy (parentheses)
squares = (x * x for x in range(1, 6))
print(sum(squares))                    # 1+4+9+16+25

Pipelines & itertools

Generators compose. Feed one into another to build a lazy pipeline where each element flows through every stage on demand — no intermediate lists. The itertools module adds a toolbox of fast, lazy building blocks like count, islice, chain, and groupby:

import itertools

# Chain lazy steps into a pipeline — nothing runs until consumed.
def read_numbers():
    for line in ["10", "-3", "20", "0", "7"]:
        yield int(line)

positives = (n for n in read_numbers() if n > 0)
doubled = (n * 2 for n in positives)

print(list(doubled))
print(list(itertools.islice(itertools.count(0, 5), 4)))   # 0,5,10,15

Recap & quick check

You can process data lazily and efficiently. Next, functions themselves become data: Module 23 — Functional Programming.