Generators and lazy evaluation

A function with yield does not run when it is called: it returns a generator, and the body only advances when someone asks for the next item. This lets you process a file larger than memory, line by line.

def read(lines):
    for line in lines:
        print("reading", line)
        yield line.upper()

g = read(["a", "b", "c"])
print("nothing has been read yet")
print(next(g))
print(list(g))

nothing has been read yet
reading a
A
reading b
reading c
['B', 'C']

A generator is single-pass

Once consumed, it is empty. A common mistake: computing the sum and then the count from the same generator.

squares = (n * n for n in range(4))
print(sum(squares))
print(sum(squares))

14
0

Generator pipeline

Chained together, generators form a pipeline in which each item goes through all the stages before the next one enters. No intermediate list is created.

from itertools import islice

def numbers():
    n = 0
    while True:
        n += 1
        yield n

evens = (n for n in numbers() if n % 2 == 0)
doubled = (n * 10 for n in evens)
print(list(islice(doubled, 5)))

[20, 40, 60, 80, 100]

The source is infinite and the program still finishes: only five items were requested. In Spark it is similar: a transformation builds the plan, and only the action (count, write, show) makes the work happen.

• For fixed-size batches, itertools.batched(it, n) (Python 3.12+) returns tuples of up to n items.