Files & Folders I/O in Python: Complete File System Operations and Directory Management Guide

Reading and writing files feels like one of the simplest things you can do in Python — until you run into encoding errors, forget to close a file handle, or accidentally overwrite something important. I’ve made most of these mistakes myself, and over time I’ve built up a set of habits and a solid mental model for how Python’s file I/O actually works. This guide covers everything from basic file reading to directory management, buffering, and safe file-handling patterns.

Opening Files: The open() Function

Everything starts with the built-in open() function, which returns a file object I can read from or write to.

f = open("notes.txt", "r")
content = f.read()
f.close()
print(content)

I almost never write it this way in real code, though, because forgetting f.close() — or having an exception occur before it’s reached — leaves the file handle open, which can leak file descriptors or leave writes unflushed to disk.

The with Statement: Why I Always Use It

Python’s with statement (a context manager) guarantees the file gets closed automatically, even if an exception is raised inside the block. This is the pattern I use for essentially every file operation I write.

with open("notes.txt", "r") as f:
    content = f.read()

print(content)  # file is already closed here, but content is still accessible

Internally, open() returns an object implementing the context manager protocol (__enter__ and __exit__). When the with block exits — normally or via an exception — Python calls f.__exit__(), which closes the file. I don’t need to remember to call .close() manually, and I don’t need to wrap things in try/finally myself.

File Modes

The second argument to open() controls how the file is opened:

The mode I’ve been burned by is "w" — it silently and immediately truncates an existing file the moment it’s opened, even before I write anything. I learned to double- and triple-check that I actually mean "w" and not "a" before running any script that opens a file for writing.

with open("output.txt", "w") as f:
    f.write("This overwrites everything that was there before.\n")

with open("output.txt", "a") as f:
    f.write("This gets appended to the end instead.\n")

Reading Files: Different Approaches

with open("notes.txt", "r") as f:
    whole_thing = f.read()          # entire file as one string

with open("notes.txt", "r") as f:
    all_lines = f.readlines()       # list of lines, each ending with '\n'

with open("notes.txt", "r") as f:
    first_line = f.readline()       # just one line

For large files, I avoid .read() or .readlines(), since both load the entire file into memory at once. Instead, I iterate over the file object directly, which reads it line by line lazily:

with open("huge_log.txt", "r") as f:
    for line in f:
        if "ERROR" in line:
            print(line.strip())

This works because file objects are themselves iterators — each iteration pulls the next line from disk without holding the rest of the file in memory, which matters a lot when processing multi-gigabyte log files.

Writing Files

with open("output.txt", "w") as f:
    f.write("First line\n")
    f.write("Second line\n")

    lines = ["Third line\n", "Fourth line\n"]
    f.writelines(lines)

write() does not add a newline automatically — I have to include \n myself, which is a common source of accidentally-concatenated output when I forget it.

Character Encoding

This is the single biggest source of file-handling bugs I’ve personally dealt with, especially when working with files containing non-ASCII characters. I always specify the encoding explicitly rather than relying on the platform default, which can differ between operating systems.

with open("notes.txt", "r", encoding="utf-8") as f:
    content = f.read()

with open("output.txt", "w", encoding="utf-8") as f:
    f.write("Café, naïve, façade — all fine with UTF-8.\n")

Without an explicit encoding="utf-8", Python uses locale.getpreferredencoding(), which might be cp1252 on some Windows setups and utf-8 on most Linux/macOS systems — meaning the exact same script can behave differently depending on where it runs. I treat encoding="utf-8" as a near-mandatory argument for any file I open in text mode.

Binary Files

For non-text files — images, PDFs, executables, serialized data — I open the file in binary mode, which reads and writes bytes objects instead of str.

with open("image.png", "rb") as f:
    data = f.read()
    print(type(data))  # <class 'bytes'>

with open("copy.png", "wb") as f:
    f.write(data)

Working with CSV, JSON, and Structured Data

For structured formats, I use Python’s dedicated standard library modules rather than manually parsing text:

import csv

with open("data.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row)

import json

with open("config.json", "r", encoding="utf-8") as f:
    config = json.load(f)

with open("config.json", "w", encoding="utf-8") as f:
    json.dump(config, f, indent=2)

I always pass newline="" when opening CSV files, per the csv module’s own documentation — otherwise, extra blank lines can appear on Windows due to how newline translation interacts with the CSV writer.

Directory (Folder) Operations

Listing Directory Contents

import os

entries = os.listdir("project")
print(entries)  # list of filenames and subdirectory names, unsorted

for entry in sorted(entries):
    print(entry)

Creating and Removing Directories

os.mkdir("new_folder")                 # single directory
os.makedirs("a/b/c", exist_ok=True)    # nested directories, safely

os.rmdir("empty_folder")               # removes only if empty

import shutil
shutil.rmtree("folder_with_stuff")     # removes a directory and everything inside it

I treat shutil.rmtree() with real caution — it deletes recursively and permanently, with no recycle bin or undo. I never call it on a path built from unchecked user input.

Copying, Moving, and Renaming

import shutil

shutil.copy("source.txt", "backup/source.txt")     # copies a single file
shutil.copytree("project", "project_backup")        # recursively copies a directory

shutil.move("old_name.txt", "new_name.txt")          # rename or move
os.rename("old_name.txt", "new_name.txt")             # rename only, simpler alternative

Modern Alternative: pathlib

I’ve increasingly moved toward pathlib.Path for file operations because it lets me chain path manipulation and I/O in a single, readable object:

from pathlib import Path

p = Path("notes.txt")
content = p.read_text(encoding="utf-8")
p.write_text("New content\n", encoding="utf-8")

for file in Path("project").rglob("*.py"):
    print(file)

read_text() and write_text() internally handle opening, encoding, and closing the file in a single call, which is genuinely convenient for simple read/write operations that don’t need line-by-line processing.

Handling Errors Gracefully

File operations fail often enough in the real world — missing files, permission issues, full disks — that I always wrap them defensively:

try:
    with open("might_not_exist.txt", "r", encoding="utf-8") as f:
        content = f.read()
except FileNotFoundError:
    print("File not found.")
except PermissionError:
    print("Permission denied.")
except UnicodeDecodeError:
    print("File is not valid UTF-8 text.")

Catching specific exceptions rather than a bare except: lets me respond appropriately to each failure mode instead of masking bugs I didn’t anticipate.

Real-World Example: Backing Up a Folder

Here’s a small automation script combining most of what I’ve covered — I’ve used something close to this for quick personal backups:

import shutil
from pathlib import Path
from datetime import datetime

def backup_folder(source, backup_root="backups"):
    source = Path(source)
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    destination = Path(backup_root) / f"{source.name}_{timestamp}"
    shutil.copytree(source, destination)
    print(f"Backed up {source} to {destination}")

backup_folder("project")

Performance and Buffering Notes

Python’s open() uses buffered I/O by default, meaning writes aren’t necessarily flushed to disk immediately — they’re batched for performance. If I need to guarantee data is written before continuing (for example, before a program might crash), I call f.flush() or use os.fsync() for a stronger guarantee:

with open("critical.log", "a", encoding="utf-8") as f:
    f.write("Important event\n")
    f.flush()

For very large files, reading in fixed-size chunks (rather than line by line, if lines aren’t meaningful — e.g., binary data) keeps memory usage predictable:

with open("large_file.bin", "rb") as f:
    while chunk := f.read(8192):
        process(chunk)

Common Mistakes I’ve Made

• Opening a file with "w" when I meant "a", silently destroying existing data.
• Forgetting encoding="utf-8", producing UnicodeDecodeError on files with non-ASCII characters, especially when moving scripts between operating systems.
• Reading an entire multi-gigabyte file into memory with .read() instead of iterating line by line.
• Not using with, leaving file handles open and occasionally hitting “too many open files” errors in long-running scripts.
• Using shutil.rmtree() carelessly on a path that wasn’t fully verified first.

FAQs

Do I need to call .close() if I use with open(...)? No — the with statement handles closing automatically, even if an exception occurs inside the block.

What’s the difference between "w" and "a" mode? "w" truncates the file immediately upon opening, discarding existing content. "a" preserves existing content and writes new data at the end.

Why do I get a UnicodeDecodeError on some files but not others? This usually happens when a file isn’t encoded the way Python assumes (often because no encoding was specified). Explicitly passing encoding="utf-8" — or the correct encoding for that specific file — resolves most of these issues.

How do I check if a file exists before opening it? Use os.path.exists() or, with pathlib, Path("file.txt").exists(). That said, I generally prefer wrapping the open() call in a try/except for FileNotFoundError, since checking first and opening after still has a small race-condition window.

Summary

File and folder I/O in Python looks simple on the surface, but reliable file handling comes down to a handful of habits: always use with, always specify an encoding for text files, choose the right mode deliberately, and prefer line-by-line or chunked reading for large files instead of loading everything into memory. Once these became second nature, file-handling bugs — the silent-data-loss kind especially — stopped showing up in my code nearly as often.

References

• Python Official Documentation — Reading and Writing Files
• Python Official Documentation — open() built-in function
• Python Official Documentation — os module
• Python Official Documentation — shutil module
• Python Official Documentation — pathlib module
• Python Official Documentation — csv module