Skip to content

Testing with pytest

What this lesson gives you

Tests are executable documentation that catch regressions before they reach production. pytest is Python's de-facto testing framework — learn fixtures, parametrize, mocking, and coverage in a single lesson.

Estimated time: 22 min read · Part: Python Tooling & Data Fundamentals

Learning Objectives

After this lesson you will be able to: (1) explain the testing pyramid and which layer to invest in first; (2) write pytest tests using plain assertions; (3) build reusable fixtures with conftest.py; (4) parametrize a test to cover multiple inputs with one function; (5) mock external dependencies so unit tests run without real network or database calls; (6) measure and enforce coverage in CI.

The Testing Pyramid

Not all tests are equal in cost or value. The testing pyramid is a mental model for balancing test investment: write many fast, cheap unit tests at the base; fewer integration tests in the middle; and only the end-to-end (E2E) tests you truly need at the top. Unit tests run in milliseconds and localize failures precisely; E2E tests are slow, brittle, and expensive to maintain. The pyramid shape reflects the optimal ratio.

       /\
      /  \    E2E / UI tests
     /    \   Slow, brittle, few
    /------\
   /        \  Integration tests
  /          \  Medium speed, moderate number
 /------------\
/              \   Unit tests
/                \  Fast, isolated, many
/------------------\

In consulting projects, teams that invert the pyramid — many slow E2E tests, almost no unit tests — suffer from a "flaky test" epidemic: tests fail intermittently due to network timeouts, timing issues, and environment differences, so engineers learn to re-run instead of fix. Shifting investment toward fast unit tests dramatically improves both CI speed and developer trust in the suite.

Why pytest Beat unittest

Python ships unittest in the standard library, but pytest became the community standard because: (1) tests are plain functions with plain assert statements — no boilerplate classes or self.assertEqual wrappers; (2) failure messages are rewritten to show actual vs expected values, not just "AssertionError"; (3) fixtures are explicit and composable rather than setUp/tearDown inheritance; (4) the plugin ecosystem (pytest-cov, pytest-asyncio, pytest-mock) is enormous. pytest also runs unittest-style tests transparently, so you can adopt it incrementally.

Anatomy of a pytest Test

pytest discovers test files matching test_*.py or *_test.py and functions or methods starting with test_. No base class required — just write a function with assert statements. When an assertion fails, pytest rewrites the expression to show exactly what the left and right sides evaluated to:

tests/test_pipeline.py
from acme_data_pipeline.pipeline import normalise_event, EventSchema
import pytest

# ── Basic unit test ───────────────────────────────────────────────
def test_normalise_strips_whitespace():
    raw = {"user_id": "  u123  ", "event": "click", "value": 42}
    result = normalise_event(raw)
    assert result["user_id"] == "u123"    # pytest rewrites assertion on failure

def test_normalise_rejects_negative_value():
    with pytest.raises(ValueError, match="value must be non-negative"):
        normalise_event({"user_id": "u1", "event": "click", "value": -1})

def test_event_schema_serialises_to_dict():
    event = EventSchema(user_id="u1", event="click", value=10)
    d = event.model_dump()
    assert d["user_id"] == "u1"
    assert d["value"] == 10
    assert "timestamp" in d    # schema auto-adds timestamp

# ── Grouping related tests in a class (no inheritance needed) ────
class TestNormaliseEdgeCases:
    def test_zero_value_is_valid(self):
        result = normalise_event({"user_id": "u1", "event": "view", "value": 0})
        assert result["value"] == 0

    def test_missing_user_id_raises(self):
        with pytest.raises(KeyError):
            normalise_event({"event": "click", "value": 5})
terminal
# Run the full suite
uv run pytest

# Run only tests in one file
uv run pytest tests/test_pipeline.py

# Run a single test by name (substring match with -k)
uv run pytest -k "test_normalise_strips"

# Verbose output: show each test name as it runs
uv run pytest -v

# Stop after first failure (fast feedback loop)
uv run pytest -x

# Show local variables on failure (very useful for debugging)
uv run pytest -l

tests/test_pipeline.py::test_normalise_strips_whitespace PASSED tests/test_pipeline.py::test_normalise_rejects_negative_value PASSED tests/test_pipeline.py::test_event_schema_serialises_to_dict PASSED 3 passed in 0.12s

Fixtures: Reusable Test State

A fixture is a decorated function that sets up (and optionally tears down) resources for tests. Fixtures replace setUp/tearDown with something more composable: a test declares which fixtures it needs as parameters, and pytest injects them automatically. Fixtures can depend on other fixtures, and conftest.py files make fixtures available across test files without explicit imports.

tests/conftest.py
import pytest
import sqlite3
from acme_data_pipeline.db import create_schema, UserRepository

# scope="session" → created once for the entire test run (expensive resources)
# scope="function" → default: fresh copy for each test (safe isolation)
# scope="module"   → one per test file

@pytest.fixture(scope="function")
def db_conn():
    """In-memory SQLite database — isolated per test, no cleanup needed."""
    conn = sqlite3.connect(":memory:")
    create_schema(conn)
    yield conn               # tests run here; everything after yield is teardown
    conn.close()

@pytest.fixture
def user_repo(db_conn):
    """Depends on db_conn — pytest resolves the dependency graph automatically."""
    return UserRepository(db_conn)

@pytest.fixture
def sample_users():
    return [
        {"user_id": "u1", "email": "ada@acme.io",    "tier": "enterprise"},
        {"user_id": "u2", "email": "grace@acme.io",  "tier": "starter"},
        {"user_id": "u3", "email": "alan@acme.io",   "tier": "enterprise"},
    ]

# ── Using fixtures in tests ──────────────────────────────────────
def test_insert_and_retrieve_user(user_repo, sample_users):
    user_repo.bulk_insert(sample_users[:1])
    user = user_repo.get("u1")
    assert user["email"] == "ada@acme.io"

def test_enterprise_users_count(user_repo, sample_users):
    user_repo.bulk_insert(sample_users)
    count = user_repo.count_by_tier("enterprise")
    assert count == 2

Fixture Scope Controls Cost vs. Isolation

The scope decision is a tradeoff. scope="function" gives perfect isolation (each test gets a clean slate) but can be slow if the fixture is expensive to create — like spinning up a database or opening an HTTP connection. scope="session" creates it once and is fast, but tests may interfere through shared state. The rule of thumb: use function scope by default; escalate to module or session only for genuinely expensive resources, and make those fixtures idempotent so test order doesn't matter.

Parametrize: Covering Multiple Inputs

@pytest.mark.parametrize runs the same test function with different input/output pairs, generating a separate test case for each. This is far better than a loop inside a test — failures identify exactly which parameter set failed, and the test count in your CI report is accurate:

tests/test_pipeline.py
import pytest
from acme_data_pipeline.pipeline import normalise_event

# Each tuple becomes a separate test case with its own PASS/FAIL line
@pytest.mark.parametrize("raw_id, expected", [
    ("  u123  ",  "u123"),     # leading/trailing whitespace
    ("U123",      "u123"),     # uppercase normalised to lower
    ("u-123",     "u-123"),    # hyphens preserved
    ("",          None),       # empty string returns None (edge case)
])
def test_user_id_normalisation(raw_id, expected):
    raw = {"user_id": raw_id, "event": "click", "value": 0}
    if expected is None:
        with pytest.raises(ValueError):
            normalise_event(raw)
    else:
        result = normalise_event(raw)
        assert result["user_id"] == expected

# Parametrize over multiple columns
@pytest.mark.parametrize("value, should_raise", [
    (0,    False),
    (100,  False),
    (-1,   True),
    (-999, True),
])
def test_value_validation(value, should_raise):
    raw = {"user_id": "u1", "event": "click", "value": value}
    if should_raise:
        with pytest.raises(ValueError):
            normalise_event(raw)
    else:
        normalise_event(raw)   # should not raise

tests/test_pipeline.py::test_user_id_normalisation[ u123 -u123] PASSED tests/test_pipeline.py::test_user_id_normalisation[U123-u123] PASSED tests/test_pipeline.py::test_user_id_normalisation[u-123-u-123] PASSED tests/test_pipeline.py::test_user_id_normalisation[-None] PASSED 8 passed in 0.07s

Mocking External Dependencies

A unit test should not make real HTTP calls or touch a real database — tests would be slow, flaky, and would break when the external service is down. unittest.mock.patch (or the pytest-mock plugin's mocker fixture) replaces the real object with a mock that you control: specify return values, raise exceptions on demand, and assert that the mock was called with the expected arguments.

tests/test_ingestor.py
from unittest.mock import patch, MagicMock
import pytest
from acme_data_pipeline.ingestor import fetch_events, IngestorError

# Patch the httpx.get that fetch_events calls internally
# The path "acme_data_pipeline.ingestor.httpx" targets the module where it is USED
@patch("acme_data_pipeline.ingestor.httpx")
def test_fetch_events_happy_path(mock_httpx):
    # Configure the mock response
    mock_response = MagicMock()
    mock_response.status_code = 200
    mock_response.json.return_value = [
        {"user_id": "u1", "event": "click", "value": 5}
    ]
    mock_httpx.get.return_value = mock_response

    events = fetch_events(api_url="https://api.example.com/events")

    assert len(events) == 1
    assert events[0]["user_id"] == "u1"
    mock_httpx.get.assert_called_once_with(
        "https://api.example.com/events",
        timeout=10.0,
        headers={"Authorization": "Bearer TOKEN"},
    )

@patch("acme_data_pipeline.ingestor.httpx")
def test_fetch_events_raises_on_500(mock_httpx):
    mock_response = MagicMock()
    mock_response.status_code = 500
    mock_response.raise_for_status.side_effect = Exception("500 Server Error")
    mock_httpx.get.return_value = mock_response

    with pytest.raises(IngestorError, match="upstream API failed"):
        fetch_events(api_url="https://api.example.com/events")

Patch Where It Is Used, Not Where It Is Defined

The most common mocking mistake: patching "httpx.get" when your module has already done import httpx, binding the name locally. You must patch the name as it appears in the module under test — "acme_data_pipeline.ingestor.httpx" — otherwise the real object still runs. This is the source of the vast majority of "the mock isn't working" bugs.

Pattern When to use
@patch("module.name") Patching at function level; mock injected as parameter
with patch(...) as m: Patching for a block inside a function; auto-restored on exit
mocker.patch (pytest-mock) Pytest fixture style; auto-restored after each test, cleaner syntax
MagicMock() Mock that supports magic methods (__len__, __iter__, etc.)
mock.return_value = x Set what the mock returns when called
mock.side_effect = exc Raise an exception when the mock is called
mock.assert_called_once_with(...) Assert the mock was called exactly once with specific args

Coverage: Measuring What You Test

Coverage measures which lines (and with branch=true, which branches) your tests execute. A high coverage percentage is a necessary but not sufficient condition for a good test suite — 100% line coverage can still miss edge cases if assertions are weak. That said, coverage is an excellent tool for finding untested code paths, and gating CI on a minimum threshold (80% is a common starting point) prevents coverage from silently degrading over time.

terminal
# Run with coverage; report missing lines in the terminal
uv run pytest --cov=src --cov-report=term-missing

# Generate HTML report for interactive browsing
uv run pytest --cov=src --cov-report=html
# open htmlcov/index.html

# Fail the build if coverage drops below threshold
uv run pytest --cov=src --cov-fail-under=80

---------- coverage: platform darwin, python 3.12 ---------- Name Stmts Miss Branch BrPart Cover ------------------------------------------------------------------------- src/acme_data_pipeline/pipeline.py 48 3 16 2 93% src/acme_data_pipeline/ingestor.py 31 1 8 1 96% src/acme_data_pipeline/db.py 22 0 6 0 100% TOTAL 101 4 30 3 95%

Consulting Lens: Test Quality vs. Coverage Theater

Clients sometimes show 95% coverage and wonder why bugs keep reaching production. Coverage only proves a line was executed — not that the output was verified. A test that calls a function but asserts nothing still contributes 100% coverage of that function. The real signal is a combination of: coverage above a meaningful threshold, assertions that would actually catch regressions, and tests that run in CI on every commit. When advising a team, probe: "Do your tests ever fail for the right reason?" If the suite hasn't caught a single real bug in the past quarter, coverage theater is likely the problem.

Knowledge check

InterviewYou have 95% test coverage but a critical bug just reached production. How is this possible, and what would you investigate?

  • It is impossible — 95% coverage means 95% of bugs are caught.
  • Coverage only measures which lines were executed, not whether assertions were strong enough to catch incorrect behavior; tests may execute code without verifying its output.
  • The remaining 5% of uncovered lines always contains all bugs.
  • Coverage above 90% guarantees all branches are tested.
Answer

Coverage only measures which lines were executed, not whether assertions were strong enough to catch incorrect behavior; tests may execute code without verifying its output.

Coverage is a necessary but not sufficient quality metric. A test that calls a function but asserts nothing about its return value contributes 100% coverage of that function while catching nothing. The bug likely lives in an edge case: a branch that is executed in tests but whose output is not verified, or a combination of inputs not covered by any parametrize case. Investigate by reviewing assertions in the tests covering the buggy code path, and add a regression test that would have caught the bug.

Knowledge check

ConceptWhy must you patch "acme_data_pipeline.ingestor.httpx" rather than "httpx.get" when mocking an HTTP call inside the ingestor module?

  • The httpx library doesn't support patching at the package level.
  • You must patch the name as it exists in the module under test — at import time, ingestor bound the name httpx in its own namespace. Patching the origin doesn't affect the already-bound local name.
  • Patching httpx.get would work; both paths are equivalent.
  • You must always patch the full dotted path starting from the test file.
Answer

You must patch the name as it exists in the module under test — at import time, ingestor bound the name httpx in its own namespace. Patching the origin doesn't affect the already-bound local name.

When Python runs import httpx inside ingestor.py, it binds the name httpx in ingestor 's namespace. Patching httpx.get replaces the function in the httpx package object, but if ingestor already holds a reference via its namespace, it may still see the original. Patching acme_data_pipeline.ingestor.httpx replaces the name in the exact namespace where the code looks it up — which is what makes the mock take effect. "Patch where it is used, not where it is defined" is the rule.

Knowledge check

PracticalA test fixture creates a database connection with scope="session". One test inserts a row, and the next test fails because it finds unexpected data. What is the root cause and fix?

  • Session-scoped fixtures are not allowed to create database connections.
  • Session scope means the fixture is shared across all tests; one test's mutations persist into the next. Fix: use scope="function" for isolation, or add explicit cleanup (transaction rollback) after each test.
  • pytest runs tests in a random order so session scope never works correctly.
  • The fix is to use scope="module" instead.
Answer

Session scope means the fixture is shared across all tests; one test's mutations persist into the next. Fix: use scope="function" for isolation, or add explicit cleanup (transaction rollback) after each test.

Session scope creates the fixture once and shares it — mutations accumulate across tests, causing order-dependent failures (the worst kind: they pass in isolation, fail in the suite). For database fixtures, either use function scope with an in-memory database (fast and perfectly isolated) or use a transaction fixture that wraps each test in a transaction and rolls it back on teardown. The second pattern lets you keep a real database connection while still achieving isolation.

Exercise

Write a pytest test module for a function parse_event_batch(raw_list) that: (1) filters out events with value < 0, (2) strips whitespace from user_id, (3) raises ValueError if the list is empty. Your test module should include: a plain assertion test, a pytest.raises test, a @pytest.mark.parametrize test covering at least 3 input cases, and a fixture providing a sample event batch.

Show solution
import pytest
from acme_data_pipeline.pipeline import parse_event_batch

@pytest.fixture
def sample_batch():
    return [
        {"user_id": " u1 ", "event": "click", "value": 5},
        {"user_id": "u2",   "event": "view",  "value": -1},   # filtered
        {"user_id": "u3",   "event": "click", "value": 0},
    ]

def test_filters_negative_values(sample_batch):
    result = parse_event_batch(sample_batch)
    assert len(result) == 2
    assert all(e["value"] >= 0 for e in result)

def test_strips_whitespace_from_user_id(sample_batch):
    result = parse_event_batch(sample_batch)
    assert result[0]["user_id"] == "u1"

def test_raises_on_empty_list():
    with pytest.raises(ValueError, match="batch must not be empty"):
        parse_event_batch([])

@pytest.mark.parametrize("value, included", [
    (10,  True),
    (0,   True),
    (-1,  False),
    (-99, False),
])
def test_negative_value_filtering(value, included):
    batch = [{"user_id": "u1", "event": "click", "value": value}]
    result = parse_event_batch(batch)
    assert (len(result) == 1) == included

Key Takeaways

  • The testing pyramid: many fast unit tests, fewer integration tests, minimal E2E tests.
  • pytest uses plain functions and assert — no boilerplate; failure messages are rewritten automatically.
  • Fixtures handle setup/teardown; scope controls isolation vs. performance.
  • conftest.py shares fixtures across files without explicit imports.
  • @pytest.mark.parametrize runs one test body over many input cases — each appears as a distinct test.
  • Mock external dependencies by patching the name in the module under test, not the definition site.
  • Coverage is a necessary but not sufficient quality signal — strong assertions matter more than line count.