Closures & Decorators¶
What this lesson gives you
How inner functions capture outer-scope state, how decorators transform callables without touching source code, and the real-world patterns – retry, rate-limiter, auth check – every production codebase relies on.
Estimated time: 35 min read · Part: Functions & the Functional Toolkit
A closure is what happens when an inner function "remembers" variables from the outer scope even after the outer function has returned. A decorator is a function that accepts a callable and returns a new callable – using the closure mechanism to wrap behaviour around the original. Together they form Python's most powerful mechanism for cross-cutting concerns: logging, caching, rate-limiting, authentication, and tracing can all be attached to any function with a single line of syntax, completely decoupled from the function's business logic.
How Closures Work: closure and Cell Objects¶
def make_adder(n):
def adder(x):
return x + n # 'n' is a free variable
return adder
add5 = make_adder(5)
add10 = make_adder(10)
print(add5(3)) # 8
print(add10(3)) # 13
# Each closure has its own cell holding a different 'n'
print(add5.__closure__) # (<cell at 0x...>,)
print(add5.__closure__[0].cell_contents) # 5
print(add10.__closure__[0].cell_contents) # 10
# Cell objects are shared: mutating nonlocal updates the cell
def counter():
count = 0
def inc():
nonlocal count
count += 1
return count
return inc
c = counter()
print(c(), c(), c()) # 1 2 3
# Inspect: the cell still holds the latest count
print(c.__closure__[0].cell_contents) # 3
A closure is a function + its environment
Formally, a closure = (function code, mapping of free variables → cells). The cells are heap-allocated, so they survive after the outer function's stack frame is gone. This is why add5 still knows n=5 even though make_adder returned long ago.
The Decorator Pattern¶
import time
import functools
def timer(fn):
@functools.wraps(fn) # preserves __name__, __doc__, __annotations__
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = fn(*args, **kwargs)
elapsed = time.perf_counter() - start
print(f"{fn.__name__} took {elapsed:.4f}s")
return result
return wrapper
@timer
def slow_sum(n):
"""Sum integers 0..n."""
return sum(range(n))
print(slow_sum(10_000_000))
# slow_sum took 0.3211s
# 49999995000000
# The @ syntax is purely shorthand:
# @timer is identical to:
# slow_sum = timer(slow_sum)
# functools.wraps matters:
print(slow_sum.__name__) # 'slow_sum' (not 'wrapper'!)
print(slow_sum.__doc__) # 'Sum integers 0..n.'
Decorator Factories — Three-Level Nesting¶
When a decorator needs its own configuration arguments, you add one more layer: a factory function that accepts the configuration and returns the decorator.
import time, functools, random
def retry(max_attempts=3, delay=1.0, exceptions=(Exception,)):
"""Decorator factory: retry on specified exceptions."""
def decorator(fn): # level 2 – receives the function
@functools.wraps(fn)
def wrapper(*args, **kwargs): # level 3 – called at runtime
last_exc = None
for attempt in range(1, max_attempts + 1):
try:
return fn(*args, **kwargs)
except exceptions as exc:
last_exc = exc
print(f"Attempt {attempt}/{max_attempts} failed: {exc}")
if attempt < max_attempts:
time.sleep(delay * attempt) # exponential-ish backoff
raise last_exc
return wrapper
return decorator # decorator factory returns decorator
# Usage
@retry(max_attempts=4, delay=0.1, exceptions=(ValueError, IOError))
def flaky_api_call(endpoint):
if random.random() < 0.7:
raise ValueError(f"Transient error on {endpoint}")
return f"Success from {endpoint}"
random.seed(99)
print(flaky_api_call("/health"))
Class-Based Decorators¶
import functools, time
class RateLimiter:
"""Allow at most `calls` calls per `period` seconds."""
def __init__(self, calls, period):
self.calls = calls
self.period = period
self.history = []
def __call__(self, fn):
@functools.wraps(fn)
def wrapper(*args, **kwargs):
now = time.monotonic()
# Purge old entries outside the window
self.history = [t for t in self.history if now - t < self.period]
if len(self.history) >= self.calls:
raise RuntimeError(
f"Rate limit exceeded: {self.calls} calls/{self.period}s"
)
self.history.append(now)
return fn(*args, **kwargs)
return wrapper
@RateLimiter(calls=3, period=1.0)
def fetch_quote(symbol):
return f"Price of {symbol}: ${round(100 + hash(symbol) % 50, 2)}"
for _ in range(3):
print(fetch_quote("AAPL"))
try:
fetch_quote("AAPL") # 4th call within 1 second
except RuntimeError as e:
print(f"Caught: {e}")
Stacking Decorators¶
import functools
def bold(fn):
@functools.wraps(fn)
def wrapper(*a, **kw): return f"<b>{fn(*a, **kw)}</b>"
return wrapper
def italic(fn):
@functools.wraps(fn)
def wrapper(*a, **kw): return f"<i>{fn(*a, **kw)}</i>"
return wrapper
@bold # applied second (outer)
@italic # applied first (inner)
def greet(name):
return f"Hello, {name}"
print(greet("Alice")) # <b><i>Hello, Alice</i></b>
# Execution order reads bottom-up for application,
# top-down for actual invocation:
# greet = bold(italic(greet))
# call: bold.wrapper → italic.wrapper → original greet
Stacking order matters
Decorators are applied bottom-up. With @auth_required above @timer, the timer wraps the already-auth-guarded function, so the timing includes the auth check. Swapping them times the raw function and auth is checked after timing starts. Think about which behaviour you want as the outermost layer.
Caching: @functools.cache vs @lru_cache¶
import functools
# @cache (Python 3.9+) is lru_cache(maxsize=None) – unbounded
@functools.cache
def fib(n):
if n < 2: return n
return fib(n-1) + fib(n-2)
print(fib(50)) # 12586269025 (instant)
print(fib.cache_info()) # hits=..., misses=51, maxsize=None, currsize=51
# @lru_cache(maxsize=N) – bounded LRU eviction
@functools.lru_cache(maxsize=128)
def heavy_query(region, year):
# simulate DB round-trip
import time; time.sleep(0.01)
return f"data_{region}_{year}"
heavy_query("APAC", 2023)
heavy_query("APAC", 2023) # cache hit
print(heavy_query.cache_info())
# CacheInfo(hits=1, misses=1, maxsize=128, currsize=1)
heavy_query.cache_clear() # manual invalidation
| Decorator | Bound? | Eviction | Thread-safe? | Best for |
|---|---|---|---|---|
@functools.cache | No | Never | Yes | Pure functions with small input space |
@lru_cache(maxsize=N) | Yes | LRU | Yes | Hot-path functions with bounded input set |
Manual dict cache | Custom | Custom | Needs lock | When you need TTL or external invalidation |
@property, @classmethod, @staticmethod¶
class Temperature:
def __init__(self, celsius):
self._celsius = celsius
@property
def celsius(self):
"""Getter – accessed like an attribute."""
return self._celsius
@celsius.setter
def celsius(self, value):
if value < -273.15:
raise ValueError(f"Below absolute zero: {value}")
self._celsius = value
@celsius.deleter
def celsius(self):
del self._celsius
@property
def fahrenheit(self):
return self._celsius * 9/5 + 32
@classmethod
def from_fahrenheit(cls, f):
"""Alternative constructor – receives the class, not an instance."""
return cls((f - 32) * 5/9)
@staticmethod
def is_valid(value):
"""Utility with no access to class or instance."""
return value >= -273.15
t = Temperature(100)
print(t.celsius) # 100
print(t.fahrenheit) # 212.0
t.celsius = 37
print(t.celsius) # 37
body = Temperature.from_fahrenheit(98.6)
print(round(body.celsius, 1)) # 37.0
print(Temperature.is_valid(-300)) # False
Story – The microservice that crashed under Black Friday load
An e-commerce platform had a pricing microservice that made 12 downstream calls per request – 4 to a tax API, 4 to a discount engine, 4 to an inventory service. A senior engineer added a single @functools.lru_cache(maxsize=1024) to the three inner functions (keyed on (product_id, region, timestamp//300) to cache for 5-minute windows). Under Black Friday load of 40k req/s, downstream call volume dropped from 480k/s to ~2k/s. The cache hit ratio was 99.6%. Zero code changes to business logic.
Analogy – A decorator is a gift wrapper
You have a gift (the function). A decorator is gift-wrap: it changes how it's presented (adds logging, timing, auth checks) without changing what's inside. Stacking decorators is wrapping in multiple layers. When someone opens it, they peel each layer from the outside in, reaching the original gift last.
Always use functools.wraps in decorators
Without @functools.wraps(fn), your wrapper has __name__ == 'wrapper', __doc__ == None, and no __annotations__. This breaks help(), Sphinx docs, pytest introspection, and Pydantic/FastAPI which inspect annotations to build request/response schemas. It's two characters of import and one line – never skip it.
Consulting lens – Decorators as governance
In client engagements, cross-cutting concerns (auth, audit logging, rate-limiting, SLA timing) are often scattered across hundreds of endpoints. Centralising them as decorators means a single code change enforces a new policy everywhere – e.g., adding GDPR audit logging to all data-access functions with @audit_log(category="PII"). This is the difference between a policy that's documented vs one that's enforced.
flowchart TD
Caller -->|"call greet('Alice')"| bold_wrapper
bold_wrapper -->|"fn(*args)"| italic_wrapper
italic_wrapper -->|"fn(*args)"| greet
greet -->|"'Hello, Alice'"| italic_wrapper
italic_wrapper -->|"'<i>Hello, Alice</i>'"| bold_wrapper
bold_wrapper -->|"'<b><i>Hello, Alice</i></b>'"| Caller Knowledge check
InterviewIf you decorate a function with @A then @B (A is above B), which wrapper is invoked first when the function is called?
- B's wrapper, then A's wrapper
- A's wrapper, then B's wrapper
- Both simultaneously
- The original function, then A, then B
Answer
A's wrapper, then B's wrapper
Decorators apply bottom-up: fn = A(B(fn)) . A's wrapper is the outermost layer, so it is invoked first. A's wrapper calls the inner function (B's wrapper), which calls the original. At call time, execution flows A → B → original → B returns → A returns.
Exercise
Exercise 4.3 · Circuit Breaker Decorator Implement a @circuit_breaker(failure_threshold=3, recovery_timeout=5) decorator factory. The circuit has three states: CLOSED (normal), OPEN (failing fast after too many errors), and HALF-OPEN (testing if the service recovered). Track consecutive failures; open the circuit when failure_threshold is reached; after recovery_timeout seconds, allow one attempt (HALF-OPEN); if it succeeds, close the circuit; if it fails, open again.
Show solution
import functools, time
def circuit_breaker(failure_threshold=3, recovery_timeout=5):
def decorator(fn):
state = {"status": "CLOSED", "failures": 0, "opened_at": None}
@functools.wraps(fn)
def wrapper(*args, **kwargs):
now = time.monotonic()
if state["status"] == "OPEN":
if now - state["opened_at"] >= recovery_timeout:
state["status"] = "HALF-OPEN"
print("Circuit HALF-OPEN: testing recovery")
else:
raise RuntimeError(f"Circuit OPEN: {fn.__name__} unavailable")
try:
result = fn(*args, **kwargs)
# Success
if state["status"] == "HALF-OPEN":
state.update(status="CLOSED", failures=0, opened_at=None)
print("Circuit CLOSED: service recovered")
else:
state["failures"] = 0
return result
except Exception as exc:
state["failures"] += 1
if state["failures"] >= failure_threshold or state["status"] == "HALF-OPEN":
state.update(status="OPEN", opened_at=time.monotonic())
print(f"Circuit OPEN after {state['failures']} failures")
raise
wrapper.state = state # expose for testing
return wrapper
return decorator
# Demo
import random
random.seed(1)
@circuit_breaker(failure_threshold=2, recovery_timeout=0.1)
def unreliable():
if random.random() < 0.8:
raise ConnectionError("timeout")
return "ok"
for i in range(6):
try:
print(f"Call {i+1}: {unreliable()}")
except Exception as e:
print(f"Call {i+1}: {e}")
time.sleep(0.05)
Key takeaways
- A closure packages a function with its free-variable cells; inspect via
__closure__[i].cell_contents. - A decorator is
fn = decorator(fn); the@syntax is identical shorthand. - Always apply
@functools.wraps(fn)inside every decorator to preserve metadata. - Decorator factories add a third nesting level: factory(config) → decorator(fn) → wrapper(*args).
- Stacking order matters: decorators apply bottom-up, execute outermost-first at call time.
@functools.cacheis unbounded (use for small input spaces);@lru_cache(maxsize=N)evicts least-recently-used entries.- Use decorators to enforce cross-cutting policies (auth, logging, retry, tracing) without touching business logic.