Skip to content

HTTP & Consuming REST APIs

What this lesson gives you

HTTP is the lingua franca of the internet. Understanding the protocol, reading status codes fluently, and writing resilient client code with timeouts, retries, and authentication are foundational skills for any backend or data role.

Estimated time: 20 min read · Part: HTTP, REST APIs & Databases

Learning Objectives

After this lesson you will be able to: (1) explain the HTTP request/response cycle and the role of methods, headers, and status codes; (2) consume REST APIs using httpx with proper timeouts and error handling; (3) implement common authentication patterns (API key, Bearer token); (4) explain idempotency and why it matters for safe retries; (5) build a simple retry-with-backoff client for production resilience.

HTTP Fundamentals

HTTP is a request/response protocol. A client sends a request with a method (GET, POST, PUT, PATCH, DELETE), a URL, headers (metadata about the request: content type, authorization, accept encoding), and an optional body (for POST/PUT/PATCH). The server replies with a status code, response headers, and a body — usually JSON for modern APIs. The protocol is stateless: each request is independent, and any session state must be explicitly managed via tokens or cookies.

REST (Representational State Transfer) is a convention for structuring HTTP APIs around resources. A resource — a user, an order, a product — lives at a URL, and you interact with it using HTTP methods that match the intended operation. REST is not a formal standard; it is a set of widely-adopted conventions that make APIs predictable and self-describing.

Request:                       Response:
─────────────────────────      ─────────────────────────
POST /orders HTTP/1.1          HTTP/1.1 201 Created
Host: api.acme.io              Content-Type: application/json
Authorization: Bearer TOKEN    Location: /orders/ord-8812
Content-Type: application/json
                               {
{ "product_id": "p42",           "id": "ord-8812",
  "quantity": 3,                 "status": "pending",
  "customer_id": "c99" }         "created_at": "2026-07-20T10:00Z"
                               }
Method Semantics Idempotent? Typical status
GET Retrieve a resource; never modify state Yes 200, 404
POST Create a new resource or trigger an action No (usually) 201, 202, 400
PUT Replace a resource entirely Yes 200, 204
PATCH Partially update a resource Yes (if designed carefully) 200, 204
DELETE Remove a resource Yes 204, 404

Consuming APIs with httpx

httpx is the modern Python HTTP client: it supports both synchronous and async usage, connection pooling, HTTP/2, and has a test transport for writing tests without real network calls. The classic requests library remains widely used but lacks async support. For new projects, prefer httpx.

consume_api.py
import httpx

# ── Basic GET with query params — always set a timeout ────────────
resp = httpx.get(
    "https://api.example.com/users",
    params={"active": "true", "limit": "50"},
    headers={"Authorization": "Bearer TOKEN"},
    timeout=10.0,   # seconds; raises httpx.TimeoutException if exceeded
)
resp.raise_for_status()   # raises httpx.HTTPStatusError on 4xx/5xx
users = resp.json()       # parse JSON body → Python dict/list

# ── POST a JSON body to create a resource ────────────────────────
created = httpx.post(
    "https://api.example.com/orders",
    json={"product_id": "p42", "quantity": 3, "customer_id": "c99"},
    headers={"Authorization": "Bearer TOKEN"},
    timeout=10.0,
)
print(created.status_code)   # 201 Created
print(created.headers["Location"])   # /orders/ord-8812

# ── PUT to fully replace a resource ──────────────────────────────
updated = httpx.put(
    "https://api.example.com/users/u1",
    json={"name": "Ada", "role": "admin", "active": True},
    headers={"Authorization": "Bearer TOKEN"},
    timeout=10.0,
)
updated.raise_for_status()

# ── DELETE ────────────────────────────────────────────────────────
resp = httpx.delete(
    "https://api.example.com/users/u99",
    headers={"Authorization": "Bearer TOKEN"},
    timeout=10.0,
)
assert resp.status_code == 204   # No Content — success, no body
Status range Meaning Common examples
2xx Success 200 OK, 201 Created, 202 Accepted, 204 No Content
3xx Redirect 301 Moved Permanently, 302 Found, 304 Not Modified
4xx Client error (caller's fault) 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 409 Conflict, 422 Unprocessable, 429 Too Many Requests
5xx Server error (their fault) 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout

Authentication Patterns

Almost every API requires authentication. The three patterns you encounter most are: API keys (a static secret string in a header or query param — simple but least secure), Bearer tokens (a JWT or opaque token from an OAuth2 flow, passed in the Authorization header), and OAuth2 client credentials (service-to-service: exchange a client ID and secret for a short-lived access token). Credentials must never be hard-coded in source code — always read from environment variables.

auth_patterns.py
import os
import httpx

# ── Pattern 1: API Key in header (e.g., Stripe, SendGrid) ─────────
API_KEY = os.environ["EXTERNAL_API_KEY"]   # never hard-code credentials

resp = httpx.get(
    "https://api.example.com/reports",
    headers={"X-API-Key": API_KEY},
    timeout=10.0,
)

# ── Pattern 2: Bearer token (OAuth2 / JWT) ────────────────────────
ACCESS_TOKEN = os.environ["ACCESS_TOKEN"]

resp = httpx.get(
    "https://api.example.com/data",
    headers={"Authorization": f"Bearer {ACCESS_TOKEN}"},
    timeout=10.0,
)

# ── Pattern 3: OAuth2 Client Credentials (service-to-service) ─────
def get_access_token(client_id: str, client_secret: str, token_url: str) -> str:
    """Exchange client credentials for a short-lived access token."""
    resp = httpx.post(
        token_url,
        data={
            "grant_type":    "client_credentials",
            "client_id":     client_id,
            "client_secret": client_secret,
            "scope":         "read:events write:orders",
        },
        timeout=10.0,
    )
    resp.raise_for_status()
    return resp.json()["access_token"]

# Use short-lived token; refresh before it expires (check "expires_in" field)
token = get_access_token(
    os.environ["CLIENT_ID"],
    os.environ["CLIENT_SECRET"],
    "https://auth.example.com/oauth/token",
)
resp = httpx.get(
    "https://api.example.com/protected",
    headers={"Authorization": f"Bearer {token}"},
    timeout=10.0,
)

Never Hard-Code Credentials in Source Code

An API key or secret committed to a Git repo can be found by anyone with access — including after the repo goes public or is compromised. Store credentials exclusively in environment variables or a secrets manager (AWS Secrets Manager, HashiCorp Vault, GitHub Actions secrets). Audit your history with tools like git-secrets or trufflehog before open-sourcing a project. This is a near-certain interview topic for any role touching external services.

Resilience: Timeouts, Retries & Idempotency

Production API clients need three behaviours that toy examples omit. Timeouts — without them, a hung upstream server hangs your process indefinitely. Retries with exponential backoff — transient failures (5xx, network resets, 429 rate limits) are normal; retry with increasing delays to avoid overwhelming a struggling server. Idempotency — an operation that can be called multiple times with the same effect is safe to retry; one that is not must use an idempotency key.

Idempotency & Safe Retries

An idempotent operation can be repeated without changing the result beyond the first call. GET, PUT, and DELETE are idempotent by design; POST generally is not (calling it twice creates two resources). This matters critically for retries: after a timeout on a POST request you don't know whether the server processed it. Retrying blindly might double-charge a customer or create duplicate records. The standard solution is an idempotency key: a unique request ID sent in a header; the server stores it and returns the original result for any duplicate with the same key, making the POST effectively idempotent from the client's view.

resilient_client.py
import time, uuid, httpx
from typing import Any

RETRYABLE_STATUS = {429, 500, 502, 503, 504}

def post_with_retry(
    url: str,
    payload: dict[str, Any],
    token: str,
    max_attempts: int = 4,
    base_delay: float = 0.5,
) -> dict[str, Any]:
    """
    POST with:
    - Idempotency key (safe retries for non-idempotent endpoint)
    - Exponential backoff on retryable errors
    - Timeout on every request
    """
    idempotency_key = str(uuid.uuid4())   # unique per logical operation
    last_exc: Exception | None = None

    for attempt in range(max_attempts):
        try:
            resp = httpx.post(
                url,
                json=payload,
                headers={
                    "Authorization": f"Bearer {token}",
                    "Idempotency-Key": idempotency_key,
                },
                timeout=10.0,
            )
            if resp.status_code in RETRYABLE_STATUS:
                delay = base_delay * (2 ** attempt)
                time.sleep(delay)
                continue
            resp.raise_for_status()
            return resp.json()
        except httpx.TimeoutException as e:
            last_exc = e
            delay = base_delay * (2 ** attempt)
            time.sleep(delay)

    raise RuntimeError(f"All {max_attempts} attempts failed") from last_exc

Consulting Lens: API Integration Reliability

A common source of production incidents at clients is an API integration that has no timeout — a slow third-party API stalls threads until the application becomes unresponsive. The fix is simple; the discovery usually happens at 2 AM during an incident. When reviewing a client's integration code, check three things immediately: are timeouts set on every call? Are 5xx errors retried with backoff? Are POST operations protected by idempotency keys where the endpoint could create duplicates? Finding and fixing these three patterns in an audit is high-value, low-effort work.

WHY HTTP is everywhere Every microservice, external SaaS integration, webhook, and mobile backend communicates over HTTP. Understanding it deeply — not just the happy path — is what separates code that works in demos from code that works in production at 3 AM.

HOW the resilient client pattern Always set timeouts. Retry on transient errors (5xx, 429) with exponential backoff. Use idempotency keys for POST operations that must not duplicate. Store credentials in environment variables, never in code.

WHERE any integration work Payment processors, CRM integrations, analytics APIs, ML inference endpoints, internal microservice calls. The resilient client pattern applies to all of them. For high-frequency integrations, also consider connection pooling via httpx's Client context manager.

Knowledge check

InterviewA payment POST times out and you don't know whether it succeeded. Why is simply retrying dangerous, and how do robust APIs solve this?

  • Retrying is always safe; HTTP deduplicates requests automatically.
  • A non-idempotent POST may have already succeeded, so retrying could double-charge; APIs use an idempotency key so the server recognises and ignores duplicate requests with the same key.
  • POST requests can never time out — only GET requests do.
  • You should switch to GET to make the payment idempotent.
Answer

A non-idempotent POST may have already succeeded, so retrying could double-charge; APIs use an idempotency key so the server recognises and ignores duplicate requests with the same key.

On a timeout you cannot tell whether the server processed the POST. Since POST is not idempotent, a retry risks performing the action twice — charging a card twice, creating duplicate orders, etc. The standard solution is an idempotency key: a UUID sent in a request header that the server stores alongside the result. Any subsequent request with the same key returns the original result without re-executing the operation. This makes the POST safe to retry while remaining logically unique.

Knowledge check

PracticalAn upstream API returns a 429 status code. What does this mean, and what should your client do?

  • 429 means the request was malformed; you should fix the payload and resubmit.
  • 429 Too Many Requests means you have exceeded a rate limit; back off and retry after a delay — check the Retry-After response header if present for the recommended wait time.
  • 429 is a server-side error; the API provider needs to fix it.
  • 429 means the resource does not exist; no retry is needed.
Answer

429 Too Many Requests means you have exceeded a rate limit; back off and retry after a delay — check the Retry-After response header if present for the recommended wait time.

429 is a rate limit response. The server is telling you: "slow down." The correct client behaviour is to stop sending requests for a period and then retry, using exponential backoff to avoid hammering the server. Many APIs include a Retry-After header specifying the exact number of seconds to wait. Never retry immediately on a 429 — you will keep hitting the rate limit and may trigger a longer ban.

Knowledge check

ConceptWhat is the difference between 401 and 403?

  • They are the same; some APIs use one or the other arbitrarily.
  • 401 Unauthorized means the client is not authenticated (no valid credentials supplied); 403 Forbidden means the client is authenticated but does not have permission for this resource.
  • 401 means the resource doesn't exist; 403 means the server crashed.
  • 403 applies only to file downloads; 401 applies to all resources.
Answer

401 Unauthorized means the client is not authenticated (no valid credentials supplied); 403 Forbidden means the client is authenticated but does not have permission for this resource.

The naming is famously confusing. 401 Unauthorized actually means "unauthenticated" — no valid credentials were provided, so retry with valid ones. 403 Forbidden means the server knows who you are (you are authenticated) but your account or role does not have permission for this specific resource or action — providing different credentials won't help. The correct client handling differs: 401 → re-authenticate; 403 → escalate permissions or display an "access denied" message.

Exercise

Write a function fetch_user_orders(user_id, token) that: (1) GETs orders from https://api.acme.io/users/{user_id}/orders; (2) sets a 10-second timeout; (3) raises a custom APIError on 4xx/5xx; (4) returns a list of order dicts from the JSON body. Then write a test using httpx's MockTransport that verifies the happy path and a 404 case without making real network calls.

Show solution
import httpx

class APIError(Exception):
    def __init__(self, status: int, detail: str) -> None:
        super().__init__(f"HTTP {status}: {detail}")
        self.status = status

def fetch_user_orders(user_id: str, token: str) -> list[dict]:
    resp = httpx.get(
        f"https://api.acme.io/users/{user_id}/orders",
        headers={"Authorization": f"Bearer {token}"},
        timeout=10.0,
    )
    if resp.is_error:
        raise APIError(resp.status_code, resp.text)
    return resp.json()

# Test with httpx MockTransport (no real network)
import pytest
from unittest.mock import MagicMock

def test_fetch_orders_happy_path(monkeypatch):
    mock_response = MagicMock()
    mock_response.is_error = False
    mock_response.json.return_value = [{"id": "ord-1", "value": 42}]
    monkeypatch.setattr("httpx.get", lambda *a, **kw: mock_response)
    orders = fetch_user_orders("u1", "token123")
    assert len(orders) == 1

def test_fetch_orders_404_raises(monkeypatch):
    mock_response = MagicMock()
    mock_response.is_error = True
    mock_response.status_code = 404
    mock_response.text = "user not found"
    monkeypatch.setattr("httpx.get", lambda *a, **kw: mock_response)
    with pytest.raises(APIError, match="HTTP 404"):
        fetch_user_orders("nonexistent", "token")

Key Takeaways

  • HTTP is request/response: method + URL + headers + body → status code + body.
  • Status codes: 2xx success, 3xx redirect, 4xx client error, 5xx server error.
  • Always set timeouts; call raise_for_status(); httpx is the modern client.
  • GET/PUT/DELETE are idempotent; use idempotency keys to make POST retries safe.
  • Retry with exponential backoff on 5xx and 429; never on 4xx (except 429).
  • Store credentials in environment variables — never in source code.