Iterator vs Iterable vs Generator in Python: Complete Iteration Protocol and Lazy Evaluation Guide

For a long time, I used for loops in Python without thinking twice about what was happening underneath. It wasn’t until I tried to build my own custom iterable class that I realized how much machinery Python quietly handles for me every time I write for x in something. In this guide, I want to untangle three terms that get used almost interchangeably but actually mean very different things: iterable, iterator, and generator.

The Big Picture

• An iterable is anything you can loop over — it knows how to produce an iterator.
• An iterator is the object that actually does the looping — it remembers where it is and produces the next value on demand.
• A generator is a special, convenient way of creating an iterator using functions (with yield) or expressions, without manually implementing the iterator protocol.

Every iterator is also an iterable, but not every iterable is an iterator. Every generator is an iterator, but not every iterator is a generator. Let’s build this up from the protocol level.

The Iterable Protocol: __iter__

An object is iterable if it implements the __iter__ method, which must return an iterator. Lists, tuples, strings, dictionaries, sets, and files are all iterables I use constantly.

my_list = [1, 2, 3]
iterator = iter(my_list)  # calls my_list.__iter__() internally
print(type(iterator))  # <class 'list_iterator'>

Calling the built-in iter() function on an iterable is exactly equivalent to calling its __iter__() method directly. This is what a for loop does behind the scenes every single time.

The Iterator Protocol: __iter__ and __next__

An object is an iterator if it implements both __iter__ (which, by convention, just returns self) and __next__, which returns the next value in the sequence and raises StopIteration when there’s nothing left.

numbers = [10, 20, 30]
it = iter(numbers)

print(next(it))  # 10
print(next(it))  # 20
print(next(it))  # 30
print(next(it))  # raises StopIteration

What a for Loop Actually Does

This was the moment it all clicked for me: a for loop is essentially syntactic sugar for a while loop built on the iterator protocol.

for item in [1, 2, 3]:
    print(item)

is roughly equivalent to:

iterable = [1, 2, 3]
iterator = iter(iterable)
while True:
    try:
        item = next(iterator)
    except StopIteration:
        break
    print(item)

Once I saw this expansion, list iteration, string iteration, dictionary iteration — everything — started making sense as the same underlying mechanism applied to different objects.

Building a Custom Iterator from Scratch

To really internalize this, I built a custom iterator that counts down from a number:

class CountDown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

for n in CountDown(3):
    print(n)

Output:

3
2
1

This works with a plain for loop because CountDown implements both __iter__ (returning itself) and __next__ (producing values and eventually raising StopIteration).

Separating Iterable from Iterator: Why It Matters

Here’s a subtlety I didn’t appreciate at first: if a class’s __iter__ returns self, then the object is both an iterable and an iterator — but that means it can only be iterated over once, because its internal state (self.current in the example above) gets exhausted.

cd = CountDown(3)
print(list(cd))  # [3, 2, 1]
print(list(cd))  # [] — already exhausted!

To support multiple independent iterations — the way a list does, where I can loop over it as many times as I want — I need to separate the iterable from the iterator, having __iter__ return a new iterator object each time:

class Countdown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        return CountdownIterator(self.start)

class CountdownIterator:
    def __init__(self, current):
        self.current = current

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

cd = Countdown(3)
print(list(cd))  # [3, 2, 1]
print(list(cd))  # [3, 2, 1] — works again, fresh iterator each time

Generators: The Easy Way to Build Iterators

Writing a class with __iter__ and __next__ every time I need custom iteration logic is verbose. Generators let me write the exact same behavior using a regular function and the yield keyword, and Python handles all the iterator protocol machinery automatically.

def countdown(start):
    current = start
    while current > 0:
        yield current
        current -= 1

for n in countdown(3):
    print(n)

This produces identical output to my hand-written class-based iterator, with a fraction of the code. Calling countdown(3) doesn’t run the function body immediately — it returns a generator object, which is itself an iterator (it has __iter__ and __next__ automatically).

gen = countdown(3)
print(type(gen))       # <class 'generator'>
print(next(gen))        # 3
print(next(gen))        # 2

How yield Actually Works Internally

When Python sees yield anywhere in a function body, it compiles that entire function differently — into a generator function. Calling it doesn’t execute any code; it just creates a generator object with a paused frame. Each call to next() resumes execution from exactly where it left off, runs until the next yield, and pauses again, preserving all local variables in between.

I find this “pause and resume” behavior is the single biggest conceptual difference between a generator and a normal function — the function’s entire local state, including the call stack position, is frozen between next() calls.

Generator Expressions

Just as list comprehensions have a generator equivalent, I can build simple generators inline using generator expressions, which use parentheses instead of square brackets:

squares = (x ** 2 for x in range(5))
print(type(squares))  # <class 'generator'>
print(list(squares))  # [0, 1, 4, 9, 16]

The critical performance difference: a list comprehension [x**2 for x in range(1_000_000)] builds the entire list in memory immediately, while the generator expression (x**2 for x in range(1_000_000)) produces values lazily, one at a time, using essentially constant memory regardless of the range size.

import sys

list_version = [x for x in range(1_000_000)]
gen_version = (x for x in range(1_000_000))

print(sys.getsizeof(list_version))  # a large number, e.g. ~8,448,728 bytes
print(sys.getsizeof(gen_version))   # a tiny number, e.g. ~200 bytes

Why Lazy Evaluation Matters

This laziness isn’t just a memory optimization trick — it fundamentally changes what’s possible. I can build generators for infinite sequences, which would be impossible with a list:

def natural_numbers():
    n = 1
    while True:
        yield n
        n += 1

nums = natural_numbers()
first_five = [next(nums) for _ in range(5)]
print(first_five)  # [1, 2, 3, 4, 5]

I also use generators heavily for processing large files line by line without loading the entire file into memory:

def read_large_file(path):
    with open(path) as f:
        for line in f:
            yield line.strip()

for line in read_large_file("huge_log.txt"):
    if "ERROR" in line:
        print(line)

yield from: Delegating to Sub-Generators

When a generator needs to yield values from another iterable, yield from avoids writing a manual loop:

def chain(*iterables):
    for it in iterables:
        yield from it

for value in chain([1, 2], (3, 4), "56"):
    print(value)

Output: 1 2 3 4 5 6 (each on its own line).

Comparing the Three, Side by Side

Common Mistakes I’ve Made

• Assuming a generator can be reused. Once exhausted, it stays exhausted — calling list() on it again returns an empty list.
• Calling len() on a generator, forgetting that generators don’t know their total length in advance since values are produced lazily.
• Storing a generator when I actually needed a list, then being surprised when a second loop over it produced nothing.
• Building a giant list in memory with a comprehension when a generator expression would have done the same job with far less memory.

FAQs

Is every generator an iterator? Yes. Every generator object automatically implements __iter__ and __next__, satisfying the iterator protocol.

Is every iterator an iterable? Yes, by protocol convention — an iterator’s __iter__ method returns itself, which technically makes it iterable, though usually only for a single pass.

Why does my generator run out after one for loop? Because generators are iterators, and iterators track their position internally — once they’ve produced StopIteration, there’s no way to “rewind” them. If you need to iterate multiple times, call the generator function again to get a fresh generator object.

When should I use a generator instead of a list? Use a generator when you’re processing a large or unbounded sequence of values and don’t need random access or multiple passes — file processing, streaming data, and infinite sequences are classic use cases.

Summary

Understanding the distinction between iterables, iterators, and generators reshaped how I think about looping in Python entirely. An iterable knows how to produce an iterator; an iterator tracks state and produces values one at a time via __next__; a generator is the simplest, most Pythonic way to build an iterator without writing a full class. Once I started reaching for generators for large or lazy data processing, my programs became noticeably more memory-efficient — and my mental model of what a for loop is actually doing became a lot clearer.

References

• Python Official Documentation — Iterator Types
• Python Official Documentation — Generators
• Python Official Documentation — Glossary: Iterable, Iterator, Generator
• PEP 255 — Simple Generators
• PEP 380 — Syntax for Delegating to a Subgenerator (yield from)