How Indentation Is Parsed in Python: Complete Block Structure and Code Formatting Implementation Guide

How Indentation is Parsed in python

The first time I showed a friend coming from Java my Python code, his first question was, “Wait, where are the curly braces?” That question is exactly why I wanted to write this article. Python doesn’t use braces or keywords like begin/end to define blocks of code — it uses indentation itself as syntax. That decision isn’t just a style preference baked into PEP 8; it’s actually enforced by the language’s grammar and its tokenizer. Once I understood how that worked under the hood, a lot of confusing IndentationError messages suddenly made complete sense.

Indentation as Syntax, Not Style

In most languages, indentation is purely cosmetic — you could write your entire program on one line (ignoring readability) and it would run identically. In Python, that’s not true. Indentation defines block boundaries: the body of an if statement, a for loop, a function, or a class is determined entirely by how many spaces precede each line.

def check_number(n):
    if n > 0:
        print("Positive")
    else:
        print("Non-positive")

Output for check_number(5):

Positive

If I shift that print("Positive") line even one space to the left or right relative to its sibling statements, I get an error immediately, not a silently wrong result — which is actually a nice safety net once you get used to it.

def broken_check(n):
    if n > 0:
        print("Positive")
         print("Definitely positive")  # extra space - this breaks

This raises:

IndentationError: unexpected indent

How the Tokenizer Actually Processes Indentation

This is the part that clicked for me once I looked at Python’s tokenizer internals. Python doesn’t hand your raw source code straight to the parser. First, it runs through a tokenizer that converts your text into a stream of tokens — names, operators, literals, and structural markers. Two of those structural markers are special: INDENT and DEDENT.

The tokenizer maintains an internal stack of indentation levels, starting at zero. As it reads each logical line, it measures the leading whitespace. If that whitespace is greater than the level on top of the stack, it pushes the new level onto the stack and emits an INDENT token. If the whitespace is less than the current level, it pops levels off the stack until it matches one, emitting a DEDENT token for each pop. If the whitespace doesn’t exactly match any level on the stack during a dedent, that’s when you get an IndentationError.

You can actually watch this happen using the tokenize module:

import tokenize
import io

code = """def greet():
    print("hi")
    if True:
        print("nested")
    print("back to one level")
"""

for tok in tokenize.generate_tokens(io.StringIO(code).readline):
    if tok.type in (tokenize.INDENT, tokenize.DEDENT, tokenize.NAME):
        print(tokenize.tok_name[tok.type], repr(tok.string))

Output (trimmed):

NAME 'def'
NAME 'greet'
INDENT '    '
NAME 'print'
NAME 'if'
INDENT '        '
NAME 'print'
DEDENT ''
NAME 'print'
DEDENT ''

Every time the block gets deeper, an INDENT token appears. Every time it steps back out, a DEDENT token appears. The Python grammar’s parser then uses these INDENT/DEDENT tokens the exact same way a C-family language’s parser uses { and } — they’re just invisible, whitespace-driven equivalents of braces.

Tabs vs Spaces

Python technically allows both tabs and spaces for indentation, but mixing them within the same block is dangerous and, since Python 3, explicitly disallowed in a way that raises an error rather than silently guessing.

def mixed_example():
	print("tab indented")
        print("space indented")

This raises:

TabError: inconsistent use of tabs and spaces in indentation

I always use 4 spaces per indentation level, exactly as PEP 8 recommends, and I configure my editor to convert the Tab key into 4 spaces automatically. This single setting has saved me from more headaches than almost any other configuration choice I’ve made.

Logical Lines vs Physical Lines

Python’s indentation rules apply to logical lines, not necessarily each physical line of text you type. You can split a logical line across multiple physical lines using parentheses, brackets, or a backslash, and indentation rules relax inside those continuations.

total = (
    10
    + 20
    + 30
)
print(total)

Output:

60

Here, the continuation lines inside the parentheses don’t need to follow strict block-indentation rules — Python simply treats everything between ( and ) as part of one logical line until the parenthesis closes.

Consistent Indentation Within a Block

Every statement within the same block must use the exact same indentation depth. You can choose 2 spaces, 4 spaces, or even 8 — Python doesn’t mandate a specific number — but whatever you pick for a block must stay consistent for every line in that block.

def example():
  x = 1
  y = 2
  return x + y

This works fine with 2-space indentation, and produces 3 when called. But mixing 2 spaces on one line and 4 on the next inside the same block breaks it:

def broken_example():
  x = 1
    y = 2
  return x + y
IndentationError: unexpected indent

Nested Blocks

Indentation stacks naturally for nested structures, and each level of nesting simply adds another INDENT token onto the tokenizer’s internal stack.

def classify_temperature(temp):
    if temp > 30:
        if temp > 40:
            print("Extremely hot")
        else:
            print("Hot")
    else:
        print("Comfortable")

classify_temperature(42)

Output:

Extremely hot

Each if/else introduces a new indentation level, and the tokenizer’s stack-based INDENT/DEDENT tracking is precisely what lets the parser figure out which else belongs to which if.

Why Python Was Designed This Way

Guido van Rossum, Python’s creator, made this decision deliberately: forcing indentation to be syntactically meaningful guarantees that a program’s visual structure and its actual logical structure can never diverge. In C or Java, it’s entirely possible (and a classic source of bugs) to have code that’s indented one way but braced another way, misleading a human reader even though the compiler interprets it correctly according to the braces. Python eliminates that entire class of bugs by making the indentation itself the source of truth.

Practical and Professional Implications

In real production codebases, indentation consistency isn’t just cosmetic; it’s enforced by linters and formatters as part of CI/CD pipelines. Tools I use regularly include:

  • black — an opinionated auto-formatter that rewrites your code to consistent 4-space indentation automatically.
  • flake8 — flags inconsistent indentation and PEP 8 violations before code review.
  • Editor .editorconfig files — I keep one in every repository to force 4-space, no-tab indentation across every contributor’s editor automatically.
# .editorconfig
[*.py]
indent_style = space
indent_size = 4

This tiny config file has prevented countless “works on my machine” indentation-related merge conflicts across teams I’ve worked with.

Best Practices

  • Always use 4 spaces per indentation level, per PEP 8.
  • Never mix tabs and spaces in the same file — configure your editor to insert spaces when you press Tab.
  • Use an auto-formatter like black so indentation consistency is enforced automatically rather than relying on manual discipline.
  • Keep nesting shallow. If you find yourself four or five levels deep, it’s often a sign the function should be broken into smaller pieces.

Common Mistakes to Avoid

A mistake I made constantly as a beginner was copy-pasting code from a website or PDF that used tabs into an editor configured for spaces, resulting in a TabError that seemed to come from nowhere. Now I always run pasted code through my formatter immediately, or use “paste and reindent” features most modern editors provide.

# Copy-pasted from a mixed-indentation source - looks fine visually
def process():
	x = 1
    y = 2  
    return x + y

Even though this might look aligned depending on your tab width setting, Python sees an actual tab character versus actual space characters, and raises a TabError.

FAQs

How many spaces should I use for indentation? PEP 8 recommends 4 spaces per level, and it’s the overwhelming community standard.

Can I use tabs instead of spaces? Technically yes, as long as you’re consistent within a block, but it’s discouraged. Mixing tabs and spaces in the same block raises a TabError in Python 3.

Why do I get IndentationError: unexpected indent? This happens when a line has more leading whitespace than the tokenizer expects, without a preceding statement (like if, def, for) that opens a new block.

Why do I get IndentationError: expected an indented block? This happens when a statement that requires a body (like if x:) is immediately followed by a line at the same or lower indentation level, meaning the block appears empty.

Does indentation affect performance? No. By the time your code is compiled to bytecode, indentation has already been converted into INDENT/DEDENT tokens and then into block boundaries in the AST. There’s no runtime cost tied to how many spaces you used.

Summary

Python’s use of indentation isn’t just a stylistic quirk — it’s a core part of the language’s grammar, implemented through INDENT and DEDENT tokens generated by the tokenizer based on a stack of indentation levels. This design choice forces visual structure and logical structure to always match, eliminating an entire category of bugs common in brace-delimited languages. Understanding this mechanism made indentation errors far less mysterious to me, and reinforced why consistent formatting, enforced through tools like black and .editorconfig, matters so much in real Python projects.

References

  • Python Language Reference, indentation rules: https://docs.python.org/3/reference/lexical_analysis.html#indentation
  • PEP 8 – Style Guide for Python Code: https://peps.python.org/pep-0008/
  • Python tokenize module documentation: https://docs.python.org/3/library/tokenize.html

Total
0
Shares

Leave a Reply

Previous Post
Simple example of Indentation in python

Simple Example of Indentation in Python: Complete Code Block and Whitespace Syntax Fundamentals Guide

Next Post
Single line, inline and multiline comments in python

Single Line, Inline and Multiline Comments in Python: Complete Code Documentation and Best Practices Guide

Related Posts