System Design Building Blocks¶

Estimated time: 22 min read · Part: Designing Python Systems at Scale

Load Balancers & Horizontal Scaling¶

A load balancer distributes incoming traffic across a pool of application servers. It is the front door of any horizontally scaled system. When you add a second server, requests still arrive at a single address; the load balancer decides which server handles each one.

Horizontal scaling adds more machines; vertical scaling upgrades the existing machine (more CPU, more RAM). Vertical scaling has a ceiling — there is only so much hardware you can put in one box. Horizontal scaling is theoretically unbounded, but it requires your application to be stateless: no local session, no local file cache, no in-memory state that another server won't share.

Common load-balancing algorithms:

• Round robin — rotate through servers in order; simple, works well when requests are uniform.
• Least connections — route to the server with the fewest active connections; better when request duration varies.
• IP hash — hash the client IP to always send the same client to the same server; useful for sticky sessions (but fragile when servers are added or removed).
• Weighted round robin — assign more traffic to more-capable servers.

The API Gateway¶

An API gateway is a specialised Layer-7 proxy that sits in front of your microservices and handles cross-cutting concerns:

• Authentication & authorisation — validate JWTs or API keys before the request reaches a service.
• Rate limiting — reject requests that exceed a per-client or per-plan quota.
• Routing — map /api/orders/* to the order service and /api/products/* to the product service.
• Request/response transformation — strip internal headers, translate protocols, aggregate responses from multiple services.
• Observability — emit logs and metrics for every inbound request in one central place.

Popular options: Kong, AWS API Gateway, Azure API Management, Nginx with Lua plugins, Traefik.

CDN for Static & Dynamic Content¶

A Content Delivery Network (CDN) is a geographically distributed cache layer. Static assets — images, CSS, JavaScript, font files — are expensive to serve from a single origin and cheap to cache at an edge node near the user. A CDN reduces latency from hundreds of milliseconds to single-digit milliseconds for cached content and offloads origin bandwidth by serving cache hits from the edge.

Modern CDNs (Cloudflare Workers, AWS Lambda@Edge) can also run code at the edge, enabling server-side rendering, A/B testing, and personalised responses without a round-trip to origin.

The CAP Theorem¶

Eric Brewer's CAP theorem states that a distributed data store can guarantee at most two of three properties:

• Consistency (C) — every read receives the most recent write or an error.
• Availability (A) — every request receives a response (though it may be stale).
• Partition tolerance (P) — the system continues operating when network partitions split nodes.

Network partitions in a real distributed system are not optional — networks fail. So partition tolerance is not something you choose; it is something you accept. The real trade-off is between C and A during a partition: do you reject requests you can't confirm are consistent, or do you answer with possibly-stale data?

Consistency Patterns¶

CAP is a blunt instrument; real systems offer a spectrum of consistency levels:

• Strong consistency / linearizability — every operation appears to take effect instantaneously at some point between its invocation and response. All reads see all prior writes. Cost: high coordination overhead, lower availability.
• Serializability — transactions execute as if in some serial order, but reads within a transaction may see a snapshot, not necessarily the absolute latest write. The gold standard for SQL databases (SERIALIZABLE isolation level).
• Eventual consistency — writes are accepted anywhere and propagate eventually; all replicas converge. There is a window during which replicas may disagree. AP systems (DynamoDB, Cassandra). Reads may be stale.
• Read-your-writes — a user always sees their own mutations but may not see others' recent writes. A practical middle ground: route a user's reads to the replica that received their write, or use a sticky session to the primary for a brief window after a write.
• Monotonic reads — once a client has read a value, it will not later read an older value. Prevents clients from seeing time run backwards.

SQL vs NoSQL — Right Tool Per Access Pattern¶

Relational databases (SQL) are the safe default. ACID transactions, powerful query planner, rich joins, schemas that enforce correctness. PostgreSQL handles most workloads up to millions of rows per table without special effort. Choose SQL unless you have a specific reason not to.

Reasons to consider NoSQL:

• Document model — hierarchical, schema-flexible data (MongoDB, Firestore); good for catalogs, user profiles, CMS.
• Wide-column store — sparse columns, time-series, large scale (Cassandra, HBase, Bigtable); good for IoT, analytics, messaging.
• Key-value store — pure lookup by key (DynamoDB, Redis); good for sessions, caches, feature flags.
• Graph database — relationships as first-class objects (Neo4j, Amazon Neptune); good for social graphs, recommendations, fraud detection.

Databases — Read Replicas & Sharding¶

Read replicas are the first scaling move for a read-heavy database. The primary (master) handles all writes. Asynchronous replication ships the write-ahead log to one or more replicas, which apply it and serve reads. Replication lag — typically milliseconds, occasionally seconds under load — means replicas may be slightly behind. For most reads (showing a product listing, rendering a dashboard) stale-by-milliseconds is fine. For reads that must see the user's own most-recent write (viewing a just-submitted form), route to the primary or use read-your-writes routing.

Sharding horizontally partitions data across multiple database nodes. Each node owns a disjoint subset of the data. Three strategies:

Connection Pooling¶

Opening a database connection is expensive: TCP handshake, authentication, server-side process spawn. A Python web application handling 500 req/s cannot open a new connection per request. A connection pool maintains a set of pre-opened, reusable connections and lends them to requests.

PgBouncer is a lightweight PostgreSQL connection pooler that sits between the application and PostgreSQL. In transaction-mode pooling, a connection is returned to the pool as soon as a transaction completes — allowing thousands of application connections to share tens of PostgreSQL connections. SQLAlchemy's built-in pool operates at the Python level:

from sqlalchemy import create_engine

engine = create_engine(
    "postgresql+psycopg2://user:pass@localhost/mydb",
    pool_size=10,          # steady-state connections kept open
    max_overflow=20,       # extra connections allowed under burst (total 30)
    pool_timeout=30,       # seconds to wait for a connection before raising
    pool_recycle=1800,     # recycle connections older than 30 min (avoids stale TCP)
    pool_pre_ping=True,    # test connection health before lending it out
)

# In practice: use a session factory, not engine directly
from sqlalchemy.orm import sessionmaker
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

# FastAPI dependency
def get_db():
    db = SessionLocal()
    try:
        yield db
    finally:
        db.close()  # returns connection to pool, does not close TCP

Storage Engines: B-Tree vs LSM-Tree¶

The physical engine under a database determines its read/write performance profile.

B-tree — PostgreSQL, MySQL InnoDB, SQLite. Data is stored in a balanced tree of fixed-size pages. Reads are fast (O(log N) page reads). Writes require in-place updates and random I/O — slower for sustained high write throughput. Ideal for OLTP workloads with mixed reads and writes.

LSM-tree (Log-Structured Merge-tree) — RocksDB, Cassandra, LevelDB, HBase. Writes go to an in-memory buffer (memtable) + append-only log. When the memtable is full, it is flushed to disk as a sorted run (SSTable). Reads merge results from multiple SSTables. Background compaction merges SSTables to reclaim space. Excellent write throughput; reads require checking multiple levels. Ideal for write-heavy workloads: event logs, IoT telemetry, time-series.

Consistent Hashing¶

When you shard a cache or database across N nodes and a node is added or removed, naive hash sharding (key % N) remaps almost every key — a cache miss storm. Consistent hashing maps both nodes and keys onto a circular hash ring. Adding or removing a node only remaps keys adjacent to it on the ring, approximately 1/N of all keys.

Virtual nodes (vnodes) assign each physical node multiple positions on the ring, improving load balance and reducing variance — used by Redis Cluster, Cassandra, and distributed caches.

import hashlib
import bisect
from typing import Optional

class ConsistentHashRing:
    """Distribute keys across nodes using consistent hashing with virtual nodes."""

    def __init__(self, nodes: list[str] = None, replicas: int = 150):
        self.replicas = replicas          # virtual nodes per physical node
        self._ring: dict[int, str] = {}   # hash -> node name
        self._sorted_keys: list[int] = [] # sorted ring positions

        for node in (nodes or []):
            self.add_node(node)

    def _hash(self, key: str) -> int:
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_node(self, node: str) -> None:
        for i in range(self.replicas):
            vnode_key = f"{node}#{i}"
            h = self._hash(vnode_key)
            self._ring[h] = node
            bisect.insort(self._sorted_keys, h)

    def remove_node(self, node: str) -> None:
        for i in range(self.replicas):
            vnode_key = f"{node}#{i}"
            h = self._hash(vnode_key)
            del self._ring[h]
            self._sorted_keys.remove(h)

    def get_node(self, key: str) -> Optional[str]:
        if not self._ring:
            return None
        h = self._hash(key)
        # Find the first ring position >= h (wrap around if past the end)
        idx = bisect.bisect_right(self._sorted_keys, h) % len(self._sorted_keys)
        return self._ring[self._sorted_keys[idx]]

# Example usage
ring = ConsistentHashRing(nodes=["redis-1", "redis-2", "redis-3"], replicas=150)

keys = ["user:1001", "product:42", "session:abc", "cart:99"]
for key in keys:
    print(f"{key:20s}  ->  {ring.get_node(key)}")

# Add a node — only ~25% of keys should remap
ring.add_node("redis-4")
remapped = sum(1 for k in keys if ring.get_node(k) != ring.get_node(k))
print(f"\nAdded redis-4. Remapped approximately 1/4 of keys.")

Rate Limiting Algorithms¶

Rate limiting protects services from excessive traffic — whether from a misbehaving client, a buggy retry loop, or a deliberate attack. Four algorithms cover most use cases:

A Redis token-bucket implementation suitable for a web application:

import time
import redis

# Lua script runs atomically on the Redis server — no race conditions
RATE_LIMIT_SCRIPT = """
local key        = KEYS[1]
local capacity   = tonumber(ARGV[1])   -- max tokens (burst size)
local refill_rate = tonumber(ARGV[2])  -- tokens added per second
local now        = tonumber(ARGV[3])   -- current timestamp (float)
local cost       = tonumber(ARGV[4])   -- tokens this request costs (usually 1)

local last_tokens = tonumber(redis.call('HGET', key, 'tokens') or capacity)
local last_time   = tonumber(redis.call('HGET', key, 'ts')     or now)

-- Refill tokens based on elapsed time
local elapsed = math.max(0, now - last_time)
local tokens  = math.min(capacity, last_tokens + elapsed * refill_rate)

if tokens >= cost then
    tokens = tokens - cost
    redis.call('HSET', key, 'tokens', tokens, 'ts', now)
    redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) + 1)
    return 1          -- allowed
else
    redis.call('HSET', key, 'tokens', tokens, 'ts', now)
    redis.call('EXPIRE', key, math.ceil(capacity / refill_rate) + 1)
    return 0          -- denied
end
"""

class TokenBucketLimiter:
    def __init__(self, redis_client: redis.Redis, capacity: int, refill_rate: float):
        self.redis = redis_client
        self.capacity = capacity        # max burst
        self.refill_rate = refill_rate  # tokens / second
        self._script = redis_client.register_script(RATE_LIMIT_SCRIPT)

    def is_allowed(self, identifier: str, cost: int = 1) -> bool:
        key = f"ratelimit:{identifier}"
        result = self._script(
            keys=[key],
            args=[self.capacity, self.refill_rate, time.time(), cost],
        )
        return bool(result)

# FastAPI middleware example
from fastapi import Request, HTTPException
from fastapi.responses import JSONResponse

r = redis.Redis(host="localhost", decode_responses=True)
limiter = TokenBucketLimiter(r, capacity=100, refill_rate=10)  # 10 req/s, burst 100

async def rate_limit_middleware(request: Request, call_next):
    client_ip = request.client.host
    if not limiter.is_allowed(client_ip):
        return JSONResponse(
            status_code=429,
            content={"error": "rate_limit_exceeded"},
            headers={"Retry-After": "1"},
        )
    return await call_next(request)

The Circuit Breaker Pattern¶

A circuit breaker wraps calls to a downstream dependency and tracks failure rate. When failures exceed a threshold, the breaker opens and subsequent calls fail immediately (fast fail) without touching the dependency. After a timeout, the breaker enters half-open state and allows a single probe request. If that succeeds, the breaker closes; if it fails, it opens again.

stateDiagram-v2
    [*] --> Closed
    Closed --> Open : failure_count >= threshold
    Open --> HalfOpen : timeout elapsed
    HalfOpen --> Closed : probe request succeeds
    HalfOpen --> Open : probe request fails

import threading
import time
from enum import Enum, auto
from functools import wraps
from typing import Callable, Any

class State(Enum):
    CLOSED   = auto()   # normal operation
    OPEN     = auto()   # failing fast, not calling downstream
    HALF_OPEN = auto()  # probing to see if downstream recovered

class CircuitBreakerOpen(Exception):
    """Raised when a call is rejected because the breaker is open."""

class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,      # failures before opening
        recovery_timeout: float = 30.0,  # seconds before half-open probe
        success_threshold: int = 1,      # successes in half-open to close
    ):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.success_threshold = success_threshold

        self._state = State.CLOSED
        self._failure_count = 0
        self._success_count = 0
        self._opened_at: float | None = None
        self._lock = threading.Lock()

    @property
    def state(self) -> State:
        with self._lock:
            if self._state == State.OPEN:
                elapsed = time.monotonic() - (self._opened_at or 0)
                if elapsed >= self.recovery_timeout:
                    self._state = State.HALF_OPEN
                    self._success_count = 0
            return self._state

    def call(self, func: Callable, *args, **kwargs) -> Any:
        current = self.state

        if current == State.OPEN:
            raise CircuitBreakerOpen(
                f"Circuit breaker is OPEN. Downstream unavailable."
            )

        try:
            result = func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as exc:
            self._on_failure()
            raise

    def _on_success(self) -> None:
        with self._lock:
            if self._state == State.HALF_OPEN:
                self._success_count += 1
                if self._success_count >= self.success_threshold:
                    self._state = State.CLOSED
                    self._failure_count = 0
            elif self._state == State.CLOSED:
                self._failure_count = max(0, self._failure_count - 1)

    def _on_failure(self) -> None:
        with self._lock:
            self._failure_count += 1
            if self._failure_count >= self.failure_threshold:
                self._state = State.OPEN
                self._opened_at = time.monotonic()
                self._failure_count = 0  # reset for next cycle

    def __call__(self, func: Callable) -> Callable:
        """Use as a decorator."""
        @wraps(func)
        def wrapper(*args, **kwargs):
            return self.call(func, *args, **kwargs)
        return wrapper

# Usage as decorator
payment_breaker = CircuitBreaker(failure_threshold=5, recovery_timeout=30)

@payment_breaker
def charge_card(amount: float, token: str) -> dict:
    import httpx
    response = httpx.post(
        "https://payment-gateway.example.com/charge",
        json={"amount": amount, "token": token},
        timeout=5.0,
    )
    response.raise_for_status()
    return response.json()

# Usage in request handler
def handle_checkout(cart_total: float, payment_token: str):
    try:
        result = charge_card(cart_total, payment_token)
        return {"status": "paid", "transaction_id": result["id"]}
    except CircuitBreakerOpen:
        # Enqueue for async retry instead of failing the user
        enqueue_payment_retry(cart_total, payment_token)
        return {"status": "pending", "message": "Payment queued for processing"}

The Saga Pattern for Distributed Transactions¶

Traditional two-phase commit (2PC) across microservices is fragile and creates tight coupling. The saga pattern replaces a distributed ACID transaction with a sequence of local transactions, each of which publishes an event or sends a command. If a step fails, compensating transactions undo prior steps.

Choreography sagas — each service emits events; downstream services listen and react. No central coordinator. Decoupled and resilient, but difficult to trace the overall flow and hard to enforce ordering.

Orchestration sagas — a central saga orchestrator sends commands to each service in sequence and listens for replies. Easy to trace, easy to add timeouts and compensations, but the orchestrator is a potential single point of failure and must itself be durable.

flowchart TD
    subgraph "Order Saga (Orchestrated)"
        O[Saga Orchestrator] -->|Reserve inventory| INV[Inventory Service]
        INV -->|Inventory reserved| O
        O -->|Charge card| PAY[Payment Service]
        PAY -->|Payment failed| O
        O -->|Release inventory| INV
    end
    style O fill:#4f6ef7,color:#fff
    style PAY fill:#e74c3c,color:#fff

Architecture Diagram: Multi-Tier System¶

flowchart LR
    subgraph Edge
        CDN[CDN / Edge Cache]
        GW[API Gateway<br/>auth · rate limit · routing]
    end

    subgraph AppTier["App Tier (3 instances)"]
        A1[App Server 1]
        A2[App Server 2]
        A3[App Server 3]
        CB[Circuit Breaker<br/>+ Token Bucket]
    end

    subgraph DataTier["Data Tier"]
        R[(Redis Cache)]
        PG[(PostgreSQL Primary)]
        RR1[(Read Replica 1)]
        RR2[(Read Replica 2)]
    end

    subgraph Downstream
        EXT[Payment Gateway]
    end

    User -->|HTTPS| CDN
    CDN -->|Cache miss| GW
    GW -->|LB round-robin| A1 & A2 & A3
    A1 & A2 & A3 --> R
    A1 & A2 & A3 -->|writes| PG
    A1 & A2 & A3 -->|reads| RR1 & RR2
    A1 & A2 & A3 --> CB --> EXT
    PG -->|replication| RR1 & RR2

System Design Components Reference¶

class CircuitBreakerWithBackoff(CircuitBreaker):
    def __init__(self, base_timeout: float = 10.0, max_timeout: float = 300.0, **kwargs):
        super().__init__(recovery_timeout=base_timeout, **kwargs)
        self._base_timeout = base_timeout
        self._max_timeout = max_timeout
        self._current_timeout = base_timeout

    def _on_failure(self) -> None:
        with self._lock:
            self._failure_count += 1
            if self._failure_count >= self.failure_threshold:
                # Double the timeout on each open, cap at max
                self._current_timeout = min(self._current_timeout * 2, self._max_timeout)
                self.recovery_timeout = self._current_timeout
                self._state = State.OPEN
                self._opened_at = time.monotonic()
                self._failure_count = 0
                print(f"Breaker OPEN. Next recovery in {self._current_timeout}s")

    def _on_success(self) -> None:
        super()._on_success()
        if self._state == State.CLOSED:
            self.reset()

    def reset(self) -> None:
        with self._lock:
            self._current_timeout = self._base_timeout
            self.recovery_timeout = self._base_timeout
            print(f"Breaker CLOSED. Timeout reset to {self._base_timeout}s")

# Test
import unittest.mock as mock

breaker = CircuitBreakerWithBackoff(failure_threshold=2, base_timeout=10.0)
failing = mock.Mock(side_effect=ConnectionError("down"))

for attempt in range(6):
    try:
        breaker.call(failing)
    except (ConnectionError, CircuitBreakerOpen) as e:
        print(f"Attempt {attempt+1}: {type(e).__name__}")
    print(f"  State: {breaker.state.name}, timeout: {breaker._current_timeout}s")