Skip to content

Files, pathlib & Context Managers

What this lesson gives you

The with statement guarantees cleanup. pathlib is the modern, object-oriented way to handle filesystem paths.

Estimated time: 22 min read · Part: Files, Errors & the Outside World

Learning objectives

  • Understand the context manager protocol (__enter__/__exit__) and why it guarantees cleanup
  • Open, read, write, and append to text and binary files correctly
  • Navigate the filesystem with pathlib.Path as objects rather than fragile strings
  • Write custom context managers using @contextmanager and the yield idiom
  • Handle large files efficiently with lazy line-by-line iteration and chunked reading

Every file you open is a resource the operating system must eventually reclaim. If your program crashes, raises an exception, or simply forgets to close a file, you leak that resource — a file descriptor — and on long-running services this adds up until the OS refuses to open any more files. The context manager pattern solves this deterministically: the with statement guarantees cleanup code runs regardless of how the block exits.

The same pattern generalizes far beyond files. Database connections, network sockets, thread locks, temporary directories, database transactions — all benefit from "open/acquire, use, close/release" expressed as a context manager. Understanding the protocol means you can apply it anywhere.

File I/O: Modes, Encoding, and Read Strategies

The built-in open() function opens a file and returns a file object. The mode argument determines whether you read, write, or append, and whether you're working with text or binary data. The encoding argument is critical for text files: always specify it explicitly. Relying on the platform default (locale.getpreferredencoding()) is a portability bug — it's UTF-8 on modern Linux/macOS but often cp1252 on Windows.

file_io.py
from pathlib import Path

# ── Writing ──────────────────────────────────────────────────────────────────
# "w" creates or truncates, "a" appends, "x" exclusive-create (fails if exists)
with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("first line\n")
    f.write("second line\n")
    f.writelines(["three\n", "four\n"])   # write iterable of strings

# ── Reading ───────────────────────────────────────────────────────────────────
# Read entire file at once — fine for small files (< few MB)
with open("notes.txt", encoding="utf-8") as f:
    contents = f.read()          # entire file as one string

with open("notes.txt", encoding="utf-8") as f:
    lines = f.readlines()        # list of strings, each with \n

# ── Lazy line iteration — best for large files ───────────────────────────────
# Never loads the whole file; processes one line at a time from the OS buffer
line_count = 0
with open("notes.txt", encoding="utf-8") as f:
    for line in f:               # f is its own iterator
        line = line.rstrip("\n")
        line_count += 1

# ── Binary files ─────────────────────────────────────────────────────────────
# Use "rb"/"wb" for images, PDFs, any non-text data — no encoding arg
with open("image.png", "rb") as f:
    header = f.read(8)           # read first 8 bytes
    f.seek(0)                    # rewind to start
    data = f.read()              # read all bytes

# ── pathlib one-liners ────────────────────────────────────────────────────────
Path("notes.txt").read_text(encoding="utf-8")    # entire file as str
Path("notes.txt").write_text("content\n", encoding="utf-8")  # write + close
Path("image.png").read_bytes()                   # entire file as bytes
Mode Operation File must exist? Text or Binary
"r" Read Yes Text (default)
"w" Write (truncate) No (creates) Text
"a" Append No (creates) Text
"x" Exclusive create No (fails if exists) Text
"r+" Read and write Yes Text
"rb" Read Yes Binary
"wb" Write (truncate) No Binary
"ab" Append No Binary

The Context Manager Protocol

A context manager is any object implementing __enter__ and __exit__. When Python executes with expr as v:, it calls expr.__enter__() and assigns the return value to v. When the block exits (normally or via exception), Python calls expr.__exit__(exc_type, exc_val, exc_tb). If __exit__ returns a truthy value, the exception is suppressed; if it returns falsy (or None), the exception propagates.

Context Manager Lifecycle
──────────────────────────────────────────────────────────────
  with open("f.txt") as f:
  │  1. __enter__() called        ← open file, return file object
  │     f = file_object
  │  2. Body executes
  │     ... your code using f ...
  │  3a. Body exits normally      ──► __exit__(None, None, None)
  │                                   f.close() called
  │  3b. Exception raised inside  ──► __exit__(exc_type, exc_val, tb)
  │      SomeError: "oops"             f.close() still called!
  │                                    if __exit__ returns False:
  │                                        exception re-raised ↑
  │                                    if __exit__ returns True:
  │                                        exception suppressed

  Without `with`:
  f = open("f.txt")
  # if an exception fires here → f.close() NEVER called → file handle leak
  f.close()

Why with beats try/finally for cleanup

You could write f = open(...); try: ...; finally: f.close() and achieve the same guarantee. The with statement is the syntactic sugar that packages this pattern into a reusable, readable protocol. More importantly, the context manager object encapsulates the cleanup — you don't have to remember to write the finally block every time. That encapsulation is the design principle: separate the "what needs cleanup" from "where it's used."

cm_class.py
# Implementing the protocol from scratch — instructive, though
# @contextmanager is simpler for most cases (see below)

class ManagedFile:
    def __init__(self, path, mode="r", encoding="utf-8"):
        self.path     = path
        self.mode     = mode
        self.encoding = encoding
        self.file     = None

    def __enter__(self):
        self.file = open(self.path, self.mode, encoding=self.encoding)
        return self.file          # this becomes the `as f` variable

    def __exit__(self, exc_type, exc_val, exc_tb):
        if self.file:
            self.file.close()
        return False              # don't suppress exceptions

with ManagedFile("notes.txt") as f:
    content = f.read()
# f.close() guaranteed, even if f.read() raised an exception

pathlib: Paths as Objects

pathlib.Path (Python 3.4+, practically universal since 3.6) represents a filesystem path as a first-class object. Instead of string manipulation (os.path.join(), os.path.split(), string slicing), you use the / operator to compose paths and attributes to decompose them. Paths are OS-aware: on Windows a Path uses backslashes internally; on POSIX it uses forward slashes.

pathlib_api.py
from pathlib import Path

# Constructing paths
base = Path("data")
report = base / "reports" / "q2_2026.csv"   # OS-correct separators
config  = Path.home() / ".config" / "app" / "settings.json"
here    = Path(__file__).parent              # directory containing this script

# Decomposing paths
p = Path("/projects/app/data/report_2026.csv")
p.name         # "report_2026.csv"     — filename with extension
p.stem         # "report_2026"         — filename without extension
p.suffix       # ".csv"                — extension including dot
p.suffixes     # ['.csv']              — list (e.g., ['.tar', '.gz'])
p.parent       # Path('/projects/app/data')
p.parents[1]   # Path('/projects/app')
p.parts        # ('/', 'projects', 'app', 'data', 'report_2026.csv')

# Testing existence and type
p.exists()
p.is_file()
p.is_dir()
p.is_symlink()

# Creating directories
(base / "output").mkdir(parents=True, exist_ok=True)
# parents=True: creates intermediate dirs
# exist_ok=True: doesn't raise if already exists

# Listing files
for f in Path("data").iterdir():         # all entries in directory
    print(f.name)

for f in Path(".").glob("**/*.py"):      # recursive glob
    print(f)

for f in Path(".").rglob("*.csv"):       # rglob = recursive glob shorthand
    print(f.relative_to(Path(".")))      # path relative to base

# Reading/writing (convenience wrappers)
text = p.read_text(encoding="utf-8")
p.write_text("new content\n", encoding="utf-8")
raw  = p.read_bytes()
p.write_bytes(b"\x89PNG\r\n")

# Renaming and deleting
p.rename(p.with_suffix(".tsv"))         # change extension
p.unlink(missing_ok=True)               # delete file, OK if missing
Path("empty_dir").rmdir()               # delete empty directory
Old style (os.path) Modern (pathlib)
os.path.join("a", "b", "c") Path("a") / "b" / "c"
os.path.basename(p) Path(p).name
os.path.dirname(p) Path(p).parent
os.path.splitext(p)[1] Path(p).suffix
os.path.exists(p) Path(p).exists()
os.makedirs(p, exist_ok=True) Path(p).mkdir(parents=True, exist_ok=True)
open(os.path.join(base, "f.txt")) open(base / "f.txt")
glob.glob("**/*.py", recursive=True) Path(".").rglob("*.py")

Custom Context Managers with @contextmanager

The contextlib.contextmanager decorator turns a generator function into a context manager. Everything before yield is the setup (__enter__); the yielded value becomes the as variable; everything after yield in the finally block is the cleanup (__exit__). This is the lightest way to write a context manager — no class required.

custom_cm.py
from contextlib import contextmanager, suppress
import time, tempfile, shutil
from pathlib import Path

# ── 1. Timer context manager ──────────────────────────────────────────────────
@contextmanager
def timer(label: str):
    start = time.perf_counter()
    try:
        yield                        # body runs here
    finally:                         # guaranteed even on exception
        elapsed = time.perf_counter() - start
        print(f"[{label}] {elapsed:.4f}s")

with timer("data load"):
    data = list(range(1_000_000))
# [data load] 0.0521s

# ── 2. Temporary directory that auto-cleans up ────────────────────────────────
@contextmanager
def temp_workspace():
    """Create a temp dir, yield its Path, then delete it on exit."""
    workspace = Path(tempfile.mkdtemp())
    try:
        yield workspace
    finally:
        shutil.rmtree(workspace, ignore_errors=True)

with temp_workspace() as ws:
    (ws / "output.txt").write_text("hello", encoding="utf-8")
    files = list(ws.iterdir())
    print(files)   # [PosixPath('/tmp/tmp.../output.txt')]
# workspace deleted here — always, even if exception was raised

# ── 3. Database transaction (pattern) ─────────────────────────────────────────
@contextmanager
def transaction(conn):
    """Commit on success, rollback on any exception."""
    try:
        yield conn
        conn.commit()
    except Exception:
        conn.rollback()
        raise                        # re-raise after cleanup

# with transaction(db_conn) as conn:
#     conn.execute("INSERT ...")
#     conn.execute("UPDATE ...")
# commits atomically; any error triggers rollback

# ── 4. contextlib.suppress — swallow specific exceptions ─────────────────────
with suppress(FileNotFoundError):
    Path("might_not_exist.txt").unlink()   # silently skipped if missing

[data load] 0.0521s

Context managers as reversible actions

Think of a context manager as a double-sided door: one side opens resources (connect, lock, open), the other side closes them (disconnect, unlock, close). The with statement guarantees you pass through both sides — even if you trip and fall in the middle. This is the RAII (Resource Acquisition Is Initialization) pattern from C++, made syntactically clean in Python. Every resource that has a setup/teardown lifecycle belongs in a context manager.

Consulting lens: file handle leaks in long-running services

A common production issue: a data processing service opens thousands of files per minute. Someone wrote f = open(...) without with, and an exception in processing skips the f.close(). Over hours, leaked file handles accumulate until the OS hits the per-process limit (ulimit -n, typically 1024 or 4096) and starts refusing open() calls with OSError: [Errno 24] Too many open files. The service crashes at midnight. Fix: always use with open(...). The rule is simple and the violation is always costly.

Knowledge check

InterviewWhat guarantee does with open(...) as f: give that a bare f = open(...) does not?

  • It makes file reads faster by buffering aggressively.
  • It guarantees f.close() runs (via __exit__) even if an exception is raised inside the block — preventing OS file handle leaks.
  • It prevents the file from being modified by other processes.
  • It loads the entire file into memory for faster access.
Answer

It guarantees f.close() runs (via __exit__) even if an exception is raised inside the block — preventing OS file handle leaks.

A context manager's exit runs deterministically when the with block exits — whether normally or via exception. For files, that means close() always executes, releasing the OS file descriptor and flushing write buffers. Manual open() / close() leaks the handle if any exception occurs between them.

Knowledge check

Concept CheckYou need to process a 10GB log file line by line. Which approach avoids loading it into memory?

  • lines = open("log.txt").readlines() then iterate.
  • content = open("log.txt").read(); content.split("\n")
  • with open("log.txt") as f: for line in f: — file objects are iterators that yield one line at a time from the OS buffer.
  • You must read the whole file first; there's no lazy line iteration in Python.
Answer

with open("log.txt") as f: for line in f: — file objects are iterators that yield one line at a time from the OS buffer.

A file object returned by open() is its own iterator: each iteration calls an internal readline() which reads from the OS's I/O buffer. Only one line is in Python memory at a time. readlines() and read().split() both load the entire file into a Python string or list first — catastrophic for 10GB files. Lazy iteration is the canonical solution for large files.

Knowledge check

GotchaWhat does the yield in a @contextmanager function represent?

  • The return value of the context manager — it replaces return.
  • The point where the with block body executes — code before yield is setup (__enter__), code in finally after yield is teardown (__exit__).
  • A lazy generator of file lines.
  • It marks the function as asynchronous.
Answer

The point where the with block body executes — code before yield is setup (__enter__), code in finally after yield is teardown (__exit__).

In a @contextmanager generator, execution suspends at yield and the with block body runs. The value yielded (if any) becomes the as variable. When the block exits — normally or via exception — execution resumes after yield . Wrapping the yield in try/finally ensures cleanup runs regardless. It's a clever use of generator suspension to implement the two-phase enter/exit protocol.