Files, pathlib & Context Managers¶
What this lesson gives you
The with statement guarantees cleanup. pathlib is the modern, object-oriented way to handle filesystem paths.
Estimated time: 22 min read · Part: Files, Errors & the Outside World
Learning objectives
- Understand the context manager protocol (
__enter__/__exit__) and why it guarantees cleanup - Open, read, write, and append to text and binary files correctly
- Navigate the filesystem with
pathlib.Pathas objects rather than fragile strings - Write custom context managers using
@contextmanagerand theyieldidiom - Handle large files efficiently with lazy line-by-line iteration and chunked reading
Every file you open is a resource the operating system must eventually reclaim. If your program crashes, raises an exception, or simply forgets to close a file, you leak that resource — a file descriptor — and on long-running services this adds up until the OS refuses to open any more files. The context manager pattern solves this deterministically: the with statement guarantees cleanup code runs regardless of how the block exits.
The same pattern generalizes far beyond files. Database connections, network sockets, thread locks, temporary directories, database transactions — all benefit from "open/acquire, use, close/release" expressed as a context manager. Understanding the protocol means you can apply it anywhere.
File I/O: Modes, Encoding, and Read Strategies¶
The built-in open() function opens a file and returns a file object. The mode argument determines whether you read, write, or append, and whether you're working with text or binary data. The encoding argument is critical for text files: always specify it explicitly. Relying on the platform default (locale.getpreferredencoding()) is a portability bug — it's UTF-8 on modern Linux/macOS but often cp1252 on Windows.
from pathlib import Path
# ── Writing ──────────────────────────────────────────────────────────────────
# "w" creates or truncates, "a" appends, "x" exclusive-create (fails if exists)
with open("notes.txt", "w", encoding="utf-8") as f:
f.write("first line\n")
f.write("second line\n")
f.writelines(["three\n", "four\n"]) # write iterable of strings
# ── Reading ───────────────────────────────────────────────────────────────────
# Read entire file at once — fine for small files (< few MB)
with open("notes.txt", encoding="utf-8") as f:
contents = f.read() # entire file as one string
with open("notes.txt", encoding="utf-8") as f:
lines = f.readlines() # list of strings, each with \n
# ── Lazy line iteration — best for large files ───────────────────────────────
# Never loads the whole file; processes one line at a time from the OS buffer
line_count = 0
with open("notes.txt", encoding="utf-8") as f:
for line in f: # f is its own iterator
line = line.rstrip("\n")
line_count += 1
# ── Binary files ─────────────────────────────────────────────────────────────
# Use "rb"/"wb" for images, PDFs, any non-text data — no encoding arg
with open("image.png", "rb") as f:
header = f.read(8) # read first 8 bytes
f.seek(0) # rewind to start
data = f.read() # read all bytes
# ── pathlib one-liners ────────────────────────────────────────────────────────
Path("notes.txt").read_text(encoding="utf-8") # entire file as str
Path("notes.txt").write_text("content\n", encoding="utf-8") # write + close
Path("image.png").read_bytes() # entire file as bytes
| Mode | Operation | File must exist? | Text or Binary |
|---|---|---|---|
"r" | Read | Yes | Text (default) |
"w" | Write (truncate) | No (creates) | Text |
"a" | Append | No (creates) | Text |
"x" | Exclusive create | No (fails if exists) | Text |
"r+" | Read and write | Yes | Text |
"rb" | Read | Yes | Binary |
"wb" | Write (truncate) | No | Binary |
"ab" | Append | No | Binary |
The Context Manager Protocol¶
A context manager is any object implementing __enter__ and __exit__. When Python executes with expr as v:, it calls expr.__enter__() and assigns the return value to v. When the block exits (normally or via exception), Python calls expr.__exit__(exc_type, exc_val, exc_tb). If __exit__ returns a truthy value, the exception is suppressed; if it returns falsy (or None), the exception propagates.
Context Manager Lifecycle
──────────────────────────────────────────────────────────────
with open("f.txt") as f:
│
│ 1. __enter__() called ← open file, return file object
│ f = file_object
│
│ 2. Body executes
│ ... your code using f ...
│
│ 3a. Body exits normally ──► __exit__(None, None, None)
│ f.close() called
│
│ 3b. Exception raised inside ──► __exit__(exc_type, exc_val, tb)
│ SomeError: "oops" f.close() still called!
│ if __exit__ returns False:
│ exception re-raised ↑
│ if __exit__ returns True:
│ exception suppressed
Without `with`:
f = open("f.txt")
# if an exception fires here → f.close() NEVER called → file handle leak
f.close()
Why with beats try/finally for cleanup
You could write f = open(...); try: ...; finally: f.close() and achieve the same guarantee. The with statement is the syntactic sugar that packages this pattern into a reusable, readable protocol. More importantly, the context manager object encapsulates the cleanup — you don't have to remember to write the finally block every time. That encapsulation is the design principle: separate the "what needs cleanup" from "where it's used."
# Implementing the protocol from scratch — instructive, though
# @contextmanager is simpler for most cases (see below)
class ManagedFile:
def __init__(self, path, mode="r", encoding="utf-8"):
self.path = path
self.mode = mode
self.encoding = encoding
self.file = None
def __enter__(self):
self.file = open(self.path, self.mode, encoding=self.encoding)
return self.file # this becomes the `as f` variable
def __exit__(self, exc_type, exc_val, exc_tb):
if self.file:
self.file.close()
return False # don't suppress exceptions
with ManagedFile("notes.txt") as f:
content = f.read()
# f.close() guaranteed, even if f.read() raised an exception
pathlib: Paths as Objects¶
pathlib.Path (Python 3.4+, practically universal since 3.6) represents a filesystem path as a first-class object. Instead of string manipulation (os.path.join(), os.path.split(), string slicing), you use the / operator to compose paths and attributes to decompose them. Paths are OS-aware: on Windows a Path uses backslashes internally; on POSIX it uses forward slashes.
from pathlib import Path
# Constructing paths
base = Path("data")
report = base / "reports" / "q2_2026.csv" # OS-correct separators
config = Path.home() / ".config" / "app" / "settings.json"
here = Path(__file__).parent # directory containing this script
# Decomposing paths
p = Path("/projects/app/data/report_2026.csv")
p.name # "report_2026.csv" — filename with extension
p.stem # "report_2026" — filename without extension
p.suffix # ".csv" — extension including dot
p.suffixes # ['.csv'] — list (e.g., ['.tar', '.gz'])
p.parent # Path('/projects/app/data')
p.parents[1] # Path('/projects/app')
p.parts # ('/', 'projects', 'app', 'data', 'report_2026.csv')
# Testing existence and type
p.exists()
p.is_file()
p.is_dir()
p.is_symlink()
# Creating directories
(base / "output").mkdir(parents=True, exist_ok=True)
# parents=True: creates intermediate dirs
# exist_ok=True: doesn't raise if already exists
# Listing files
for f in Path("data").iterdir(): # all entries in directory
print(f.name)
for f in Path(".").glob("**/*.py"): # recursive glob
print(f)
for f in Path(".").rglob("*.csv"): # rglob = recursive glob shorthand
print(f.relative_to(Path("."))) # path relative to base
# Reading/writing (convenience wrappers)
text = p.read_text(encoding="utf-8")
p.write_text("new content\n", encoding="utf-8")
raw = p.read_bytes()
p.write_bytes(b"\x89PNG\r\n")
# Renaming and deleting
p.rename(p.with_suffix(".tsv")) # change extension
p.unlink(missing_ok=True) # delete file, OK if missing
Path("empty_dir").rmdir() # delete empty directory
Old style (os.path) | Modern (pathlib) |
|---|---|
os.path.join("a", "b", "c") | Path("a") / "b" / "c" |
os.path.basename(p) | Path(p).name |
os.path.dirname(p) | Path(p).parent |
os.path.splitext(p)[1] | Path(p).suffix |
os.path.exists(p) | Path(p).exists() |
os.makedirs(p, exist_ok=True) | Path(p).mkdir(parents=True, exist_ok=True) |
open(os.path.join(base, "f.txt")) | open(base / "f.txt") |
glob.glob("**/*.py", recursive=True) | Path(".").rglob("*.py") |
Custom Context Managers with @contextmanager¶
The contextlib.contextmanager decorator turns a generator function into a context manager. Everything before yield is the setup (__enter__); the yielded value becomes the as variable; everything after yield in the finally block is the cleanup (__exit__). This is the lightest way to write a context manager — no class required.
from contextlib import contextmanager, suppress
import time, tempfile, shutil
from pathlib import Path
# ── 1. Timer context manager ──────────────────────────────────────────────────
@contextmanager
def timer(label: str):
start = time.perf_counter()
try:
yield # body runs here
finally: # guaranteed even on exception
elapsed = time.perf_counter() - start
print(f"[{label}] {elapsed:.4f}s")
with timer("data load"):
data = list(range(1_000_000))
# [data load] 0.0521s
# ── 2. Temporary directory that auto-cleans up ────────────────────────────────
@contextmanager
def temp_workspace():
"""Create a temp dir, yield its Path, then delete it on exit."""
workspace = Path(tempfile.mkdtemp())
try:
yield workspace
finally:
shutil.rmtree(workspace, ignore_errors=True)
with temp_workspace() as ws:
(ws / "output.txt").write_text("hello", encoding="utf-8")
files = list(ws.iterdir())
print(files) # [PosixPath('/tmp/tmp.../output.txt')]
# workspace deleted here — always, even if exception was raised
# ── 3. Database transaction (pattern) ─────────────────────────────────────────
@contextmanager
def transaction(conn):
"""Commit on success, rollback on any exception."""
try:
yield conn
conn.commit()
except Exception:
conn.rollback()
raise # re-raise after cleanup
# with transaction(db_conn) as conn:
# conn.execute("INSERT ...")
# conn.execute("UPDATE ...")
# commits atomically; any error triggers rollback
# ── 4. contextlib.suppress — swallow specific exceptions ─────────────────────
with suppress(FileNotFoundError):
Path("might_not_exist.txt").unlink() # silently skipped if missing
[data load] 0.0521s
Context managers as reversible actions
Think of a context manager as a double-sided door: one side opens resources (connect, lock, open), the other side closes them (disconnect, unlock, close). The with statement guarantees you pass through both sides — even if you trip and fall in the middle. This is the RAII (Resource Acquisition Is Initialization) pattern from C++, made syntactically clean in Python. Every resource that has a setup/teardown lifecycle belongs in a context manager.
Consulting lens: file handle leaks in long-running services
A common production issue: a data processing service opens thousands of files per minute. Someone wrote f = open(...) without with, and an exception in processing skips the f.close(). Over hours, leaked file handles accumulate until the OS hits the per-process limit (ulimit -n, typically 1024 or 4096) and starts refusing open() calls with OSError: [Errno 24] Too many open files. The service crashes at midnight. Fix: always use with open(...). The rule is simple and the violation is always costly.
Knowledge check
InterviewWhat guarantee does with open(...) as f: give that a bare f = open(...) does not?
- It makes file reads faster by buffering aggressively.
- It guarantees
f.close()runs (via__exit__) even if an exception is raised inside the block — preventing OS file handle leaks. - It prevents the file from being modified by other processes.
- It loads the entire file into memory for faster access.
Answer
It guarantees f.close() runs (via __exit__) even if an exception is raised inside the block — preventing OS file handle leaks.
A context manager's exit runs deterministically when the with block exits — whether normally or via exception. For files, that means close() always executes, releasing the OS file descriptor and flushing write buffers. Manual open() / close() leaks the handle if any exception occurs between them.
Knowledge check
Concept CheckYou need to process a 10GB log file line by line. Which approach avoids loading it into memory?
lines = open("log.txt").readlines()then iterate.content = open("log.txt").read(); content.split("\n")with open("log.txt") as f: for line in f:— file objects are iterators that yield one line at a time from the OS buffer.- You must read the whole file first; there's no lazy line iteration in Python.
Answer
with open("log.txt") as f: for line in f: — file objects are iterators that yield one line at a time from the OS buffer.
A file object returned by open() is its own iterator: each iteration calls an internal readline() which reads from the OS's I/O buffer. Only one line is in Python memory at a time. readlines() and read().split() both load the entire file into a Python string or list first — catastrophic for 10GB files. Lazy iteration is the canonical solution for large files.
Knowledge check
GotchaWhat does the yield in a @contextmanager function represent?
- The return value of the context manager — it replaces
return. - The point where the
withblock body executes — code before yield is setup (__enter__), code infinallyafter yield is teardown (__exit__). - A lazy generator of file lines.
- It marks the function as asynchronous.
Answer
The point where the with block body executes — code before yield is setup (__enter__), code in finally after yield is teardown (__exit__).
In a @contextmanager generator, execution suspends at yield and the with block body runs. The value yielded (if any) becomes the as variable. When the block exits — normally or via exception — execution resumes after yield . Wrapping the yield in try/finally ensures cleanup runs regardless. It's a clever use of generator suspension to implement the two-phase enter/exit protocol.