Testing with pytest¶
What this lesson gives you
Tests are executable documentation that catch regressions before they reach production. pytest is Python's de-facto testing framework — learn fixtures, parametrize, mocking, and coverage in a single lesson.
Estimated time: 22 min read · Part: Python Tooling & Data Fundamentals
Learning Objectives
After this lesson you will be able to: (1) explain the testing pyramid and which layer to invest in first; (2) write pytest tests using plain assertions; (3) build reusable fixtures with conftest.py; (4) parametrize a test to cover multiple inputs with one function; (5) mock external dependencies so unit tests run without real network or database calls; (6) measure and enforce coverage in CI.
The Testing Pyramid¶
Not all tests are equal in cost or value. The testing pyramid is a mental model for balancing test investment: write many fast, cheap unit tests at the base; fewer integration tests in the middle; and only the end-to-end (E2E) tests you truly need at the top. Unit tests run in milliseconds and localize failures precisely; E2E tests are slow, brittle, and expensive to maintain. The pyramid shape reflects the optimal ratio.
/\
/ \ E2E / UI tests
/ \ Slow, brittle, few
/------\
/ \ Integration tests
/ \ Medium speed, moderate number
/------------\
/ \ Unit tests
/ \ Fast, isolated, many
/------------------\
In consulting projects, teams that invert the pyramid — many slow E2E tests, almost no unit tests — suffer from a "flaky test" epidemic: tests fail intermittently due to network timeouts, timing issues, and environment differences, so engineers learn to re-run instead of fix. Shifting investment toward fast unit tests dramatically improves both CI speed and developer trust in the suite.
Why pytest Beat unittest
Python ships unittest in the standard library, but pytest became the community standard because: (1) tests are plain functions with plain assert statements — no boilerplate classes or self.assertEqual wrappers; (2) failure messages are rewritten to show actual vs expected values, not just "AssertionError"; (3) fixtures are explicit and composable rather than setUp/tearDown inheritance; (4) the plugin ecosystem (pytest-cov, pytest-asyncio, pytest-mock) is enormous. pytest also runs unittest-style tests transparently, so you can adopt it incrementally.
Anatomy of a pytest Test¶
pytest discovers test files matching test_*.py or *_test.py and functions or methods starting with test_. No base class required — just write a function with assert statements. When an assertion fails, pytest rewrites the expression to show exactly what the left and right sides evaluated to:
from acme_data_pipeline.pipeline import normalise_event, EventSchema
import pytest
# ── Basic unit test ───────────────────────────────────────────────
def test_normalise_strips_whitespace():
raw = {"user_id": " u123 ", "event": "click", "value": 42}
result = normalise_event(raw)
assert result["user_id"] == "u123" # pytest rewrites assertion on failure
def test_normalise_rejects_negative_value():
with pytest.raises(ValueError, match="value must be non-negative"):
normalise_event({"user_id": "u1", "event": "click", "value": -1})
def test_event_schema_serialises_to_dict():
event = EventSchema(user_id="u1", event="click", value=10)
d = event.model_dump()
assert d["user_id"] == "u1"
assert d["value"] == 10
assert "timestamp" in d # schema auto-adds timestamp
# ── Grouping related tests in a class (no inheritance needed) ────
class TestNormaliseEdgeCases:
def test_zero_value_is_valid(self):
result = normalise_event({"user_id": "u1", "event": "view", "value": 0})
assert result["value"] == 0
def test_missing_user_id_raises(self):
with pytest.raises(KeyError):
normalise_event({"event": "click", "value": 5})
# Run the full suite
uv run pytest
# Run only tests in one file
uv run pytest tests/test_pipeline.py
# Run a single test by name (substring match with -k)
uv run pytest -k "test_normalise_strips"
# Verbose output: show each test name as it runs
uv run pytest -v
# Stop after first failure (fast feedback loop)
uv run pytest -x
# Show local variables on failure (very useful for debugging)
uv run pytest -l
tests/test_pipeline.py::test_normalise_strips_whitespace PASSED tests/test_pipeline.py::test_normalise_rejects_negative_value PASSED tests/test_pipeline.py::test_event_schema_serialises_to_dict PASSED 3 passed in 0.12s
Fixtures: Reusable Test State¶
A fixture is a decorated function that sets up (and optionally tears down) resources for tests. Fixtures replace setUp/tearDown with something more composable: a test declares which fixtures it needs as parameters, and pytest injects them automatically. Fixtures can depend on other fixtures, and conftest.py files make fixtures available across test files without explicit imports.
import pytest
import sqlite3
from acme_data_pipeline.db import create_schema, UserRepository
# scope="session" → created once for the entire test run (expensive resources)
# scope="function" → default: fresh copy for each test (safe isolation)
# scope="module" → one per test file
@pytest.fixture(scope="function")
def db_conn():
"""In-memory SQLite database — isolated per test, no cleanup needed."""
conn = sqlite3.connect(":memory:")
create_schema(conn)
yield conn # tests run here; everything after yield is teardown
conn.close()
@pytest.fixture
def user_repo(db_conn):
"""Depends on db_conn — pytest resolves the dependency graph automatically."""
return UserRepository(db_conn)
@pytest.fixture
def sample_users():
return [
{"user_id": "u1", "email": "ada@acme.io", "tier": "enterprise"},
{"user_id": "u2", "email": "grace@acme.io", "tier": "starter"},
{"user_id": "u3", "email": "alan@acme.io", "tier": "enterprise"},
]
# ── Using fixtures in tests ──────────────────────────────────────
def test_insert_and_retrieve_user(user_repo, sample_users):
user_repo.bulk_insert(sample_users[:1])
user = user_repo.get("u1")
assert user["email"] == "ada@acme.io"
def test_enterprise_users_count(user_repo, sample_users):
user_repo.bulk_insert(sample_users)
count = user_repo.count_by_tier("enterprise")
assert count == 2
Fixture Scope Controls Cost vs. Isolation
The scope decision is a tradeoff. scope="function" gives perfect isolation (each test gets a clean slate) but can be slow if the fixture is expensive to create — like spinning up a database or opening an HTTP connection. scope="session" creates it once and is fast, but tests may interfere through shared state. The rule of thumb: use function scope by default; escalate to module or session only for genuinely expensive resources, and make those fixtures idempotent so test order doesn't matter.
Parametrize: Covering Multiple Inputs¶
@pytest.mark.parametrize runs the same test function with different input/output pairs, generating a separate test case for each. This is far better than a loop inside a test — failures identify exactly which parameter set failed, and the test count in your CI report is accurate:
import pytest
from acme_data_pipeline.pipeline import normalise_event
# Each tuple becomes a separate test case with its own PASS/FAIL line
@pytest.mark.parametrize("raw_id, expected", [
(" u123 ", "u123"), # leading/trailing whitespace
("U123", "u123"), # uppercase normalised to lower
("u-123", "u-123"), # hyphens preserved
("", None), # empty string returns None (edge case)
])
def test_user_id_normalisation(raw_id, expected):
raw = {"user_id": raw_id, "event": "click", "value": 0}
if expected is None:
with pytest.raises(ValueError):
normalise_event(raw)
else:
result = normalise_event(raw)
assert result["user_id"] == expected
# Parametrize over multiple columns
@pytest.mark.parametrize("value, should_raise", [
(0, False),
(100, False),
(-1, True),
(-999, True),
])
def test_value_validation(value, should_raise):
raw = {"user_id": "u1", "event": "click", "value": value}
if should_raise:
with pytest.raises(ValueError):
normalise_event(raw)
else:
normalise_event(raw) # should not raise
tests/test_pipeline.py::test_user_id_normalisation[ u123 -u123] PASSED tests/test_pipeline.py::test_user_id_normalisation[U123-u123] PASSED tests/test_pipeline.py::test_user_id_normalisation[u-123-u-123] PASSED tests/test_pipeline.py::test_user_id_normalisation[-None] PASSED 8 passed in 0.07s
Mocking External Dependencies¶
A unit test should not make real HTTP calls or touch a real database — tests would be slow, flaky, and would break when the external service is down. unittest.mock.patch (or the pytest-mock plugin's mocker fixture) replaces the real object with a mock that you control: specify return values, raise exceptions on demand, and assert that the mock was called with the expected arguments.
from unittest.mock import patch, MagicMock
import pytest
from acme_data_pipeline.ingestor import fetch_events, IngestorError
# Patch the httpx.get that fetch_events calls internally
# The path "acme_data_pipeline.ingestor.httpx" targets the module where it is USED
@patch("acme_data_pipeline.ingestor.httpx")
def test_fetch_events_happy_path(mock_httpx):
# Configure the mock response
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = [
{"user_id": "u1", "event": "click", "value": 5}
]
mock_httpx.get.return_value = mock_response
events = fetch_events(api_url="https://api.example.com/events")
assert len(events) == 1
assert events[0]["user_id"] == "u1"
mock_httpx.get.assert_called_once_with(
"https://api.example.com/events",
timeout=10.0,
headers={"Authorization": "Bearer TOKEN"},
)
@patch("acme_data_pipeline.ingestor.httpx")
def test_fetch_events_raises_on_500(mock_httpx):
mock_response = MagicMock()
mock_response.status_code = 500
mock_response.raise_for_status.side_effect = Exception("500 Server Error")
mock_httpx.get.return_value = mock_response
with pytest.raises(IngestorError, match="upstream API failed"):
fetch_events(api_url="https://api.example.com/events")
Patch Where It Is Used, Not Where It Is Defined
The most common mocking mistake: patching "httpx.get" when your module has already done import httpx, binding the name locally. You must patch the name as it appears in the module under test — "acme_data_pipeline.ingestor.httpx" — otherwise the real object still runs. This is the source of the vast majority of "the mock isn't working" bugs.
| Pattern | When to use |
|---|---|
@patch("module.name") | Patching at function level; mock injected as parameter |
with patch(...) as m: | Patching for a block inside a function; auto-restored on exit |
mocker.patch (pytest-mock) | Pytest fixture style; auto-restored after each test, cleaner syntax |
MagicMock() | Mock that supports magic methods (__len__, __iter__, etc.) |
mock.return_value = x | Set what the mock returns when called |
mock.side_effect = exc | Raise an exception when the mock is called |
mock.assert_called_once_with(...) | Assert the mock was called exactly once with specific args |
Coverage: Measuring What You Test¶
Coverage measures which lines (and with branch=true, which branches) your tests execute. A high coverage percentage is a necessary but not sufficient condition for a good test suite — 100% line coverage can still miss edge cases if assertions are weak. That said, coverage is an excellent tool for finding untested code paths, and gating CI on a minimum threshold (80% is a common starting point) prevents coverage from silently degrading over time.
# Run with coverage; report missing lines in the terminal
uv run pytest --cov=src --cov-report=term-missing
# Generate HTML report for interactive browsing
uv run pytest --cov=src --cov-report=html
# open htmlcov/index.html
# Fail the build if coverage drops below threshold
uv run pytest --cov=src --cov-fail-under=80
---------- coverage: platform darwin, python 3.12 ---------- Name Stmts Miss Branch BrPart Cover ------------------------------------------------------------------------- src/acme_data_pipeline/pipeline.py 48 3 16 2 93% src/acme_data_pipeline/ingestor.py 31 1 8 1 96% src/acme_data_pipeline/db.py 22 0 6 0 100% TOTAL 101 4 30 3 95%
Consulting Lens: Test Quality vs. Coverage Theater
Clients sometimes show 95% coverage and wonder why bugs keep reaching production. Coverage only proves a line was executed — not that the output was verified. A test that calls a function but asserts nothing still contributes 100% coverage of that function. The real signal is a combination of: coverage above a meaningful threshold, assertions that would actually catch regressions, and tests that run in CI on every commit. When advising a team, probe: "Do your tests ever fail for the right reason?" If the suite hasn't caught a single real bug in the past quarter, coverage theater is likely the problem.
Knowledge check
InterviewYou have 95% test coverage but a critical bug just reached production. How is this possible, and what would you investigate?
- It is impossible — 95% coverage means 95% of bugs are caught.
- Coverage only measures which lines were executed, not whether assertions were strong enough to catch incorrect behavior; tests may execute code without verifying its output.
- The remaining 5% of uncovered lines always contains all bugs.
- Coverage above 90% guarantees all branches are tested.
Answer
Coverage only measures which lines were executed, not whether assertions were strong enough to catch incorrect behavior; tests may execute code without verifying its output.
Coverage is a necessary but not sufficient quality metric. A test that calls a function but asserts nothing about its return value contributes 100% coverage of that function while catching nothing. The bug likely lives in an edge case: a branch that is executed in tests but whose output is not verified, or a combination of inputs not covered by any parametrize case. Investigate by reviewing assertions in the tests covering the buggy code path, and add a regression test that would have caught the bug.
Knowledge check
ConceptWhy must you patch "acme_data_pipeline.ingestor.httpx" rather than "httpx.get" when mocking an HTTP call inside the ingestor module?
- The httpx library doesn't support patching at the package level.
- You must patch the name as it exists in the module under test — at import time,
ingestorbound the namehttpxin its own namespace. Patching the origin doesn't affect the already-bound local name. - Patching
httpx.getwould work; both paths are equivalent. - You must always patch the full dotted path starting from the test file.
Answer
You must patch the name as it exists in the module under test — at import time, ingestor bound the name httpx in its own namespace. Patching the origin doesn't affect the already-bound local name.
When Python runs import httpx inside ingestor.py, it binds the name httpx in ingestor 's namespace. Patching httpx.get replaces the function in the httpx package object, but if ingestor already holds a reference via its namespace, it may still see the original. Patching acme_data_pipeline.ingestor.httpx replaces the name in the exact namespace where the code looks it up — which is what makes the mock take effect. "Patch where it is used, not where it is defined" is the rule.
Knowledge check
PracticalA test fixture creates a database connection with scope="session". One test inserts a row, and the next test fails because it finds unexpected data. What is the root cause and fix?
- Session-scoped fixtures are not allowed to create database connections.
- Session scope means the fixture is shared across all tests; one test's mutations persist into the next. Fix: use
scope="function"for isolation, or add explicit cleanup (transaction rollback) after each test. - pytest runs tests in a random order so session scope never works correctly.
- The fix is to use
scope="module"instead.
Answer
Session scope means the fixture is shared across all tests; one test's mutations persist into the next. Fix: use scope="function" for isolation, or add explicit cleanup (transaction rollback) after each test.
Session scope creates the fixture once and shares it — mutations accumulate across tests, causing order-dependent failures (the worst kind: they pass in isolation, fail in the suite). For database fixtures, either use function scope with an in-memory database (fast and perfectly isolated) or use a transaction fixture that wraps each test in a transaction and rolls it back on teardown. The second pattern lets you keep a real database connection while still achieving isolation.
Exercise
Write a pytest test module for a function parse_event_batch(raw_list) that: (1) filters out events with value < 0, (2) strips whitespace from user_id, (3) raises ValueError if the list is empty. Your test module should include: a plain assertion test, a pytest.raises test, a @pytest.mark.parametrize test covering at least 3 input cases, and a fixture providing a sample event batch.
Show solution
import pytest
from acme_data_pipeline.pipeline import parse_event_batch
@pytest.fixture
def sample_batch():
return [
{"user_id": " u1 ", "event": "click", "value": 5},
{"user_id": "u2", "event": "view", "value": -1}, # filtered
{"user_id": "u3", "event": "click", "value": 0},
]
def test_filters_negative_values(sample_batch):
result = parse_event_batch(sample_batch)
assert len(result) == 2
assert all(e["value"] >= 0 for e in result)
def test_strips_whitespace_from_user_id(sample_batch):
result = parse_event_batch(sample_batch)
assert result[0]["user_id"] == "u1"
def test_raises_on_empty_list():
with pytest.raises(ValueError, match="batch must not be empty"):
parse_event_batch([])
@pytest.mark.parametrize("value, included", [
(10, True),
(0, True),
(-1, False),
(-99, False),
])
def test_negative_value_filtering(value, included):
batch = [{"user_id": "u1", "event": "click", "value": value}]
result = parse_event_batch(batch)
assert (len(result) == 1) == included
Key Takeaways
- The testing pyramid: many fast unit tests, fewer integration tests, minimal E2E tests.
- pytest uses plain functions and
assert— no boilerplate; failure messages are rewritten automatically. - Fixtures handle setup/teardown; scope controls isolation vs. performance.
conftest.pyshares fixtures across files without explicit imports.@pytest.mark.parametrizeruns one test body over many input cases — each appears as a distinct test.- Mock external dependencies by patching the name in the module under test, not the definition site.
- Coverage is a necessary but not sufficient quality signal — strong assertions matter more than line count.