Anatomy of a Pipeline¶

What this lesson gives you

Stages, jobs, runners, artifacts, environments — the universal building blocks that every CI/CD tool shares under different names. Understanding the vocabulary lets you read and reason about any pipeline configuration.

Estimated time: 18 min read · Part: CI/CD Foundations

Learning Objectives

After this lesson you will be able to: (1) define and distinguish triggers, stages, jobs, steps, runners, artifacts, caches, environments, and secrets; (2) explain why pipelines order stages cheap-to-expensive; (3) read a generic pipeline diagram and identify each component; (4) describe how artifacts flow between stages; (5) explain the role of secrets management in a pipeline.

The Universal Pipeline Vocabulary¶

Every CI/CD tool — GitHub Actions, GitLab CI, Jenkins, Azure Pipelines — implements the same conceptual model under different names. A pipeline is an automated sequence triggered by an event. It contains stages (logical phases) that run in order. Each stage contains one or more jobs (units of work) that can run in parallel within the stage. Each job runs on a runner (a machine or container) and contains steps (individual commands or reusable actions). Jobs produce artifacts (files) that can be passed to later stages.

flowchart LR
    A[Trigger<br/>Push / PR / Tag] --> B[Stage 1: Verify<br/>Lint · Format · Type-check]
    B --> C[Stage 2: Test<br/>Unit · Integration]
    C --> D[Stage 3: Build<br/>Docker Image]
    D --> E[Stage 4: Scan<br/>CVE · SAST]
    E --> F{Branch?}
    F -->|main| G[Stage 5: Deploy Staging]
    G --> H[Stage 6: E2E Tests]
    H --> I[Approval Gate]
    I --> J[Stage 7: Deploy Production]
    F -->|feature| K[Report on PR]

Concept	What it is	GitHub Actions name	GitLab CI name
Trigger	Event that starts the pipeline: push, PR, tag, schedule, manual	`on:`	`rules:`
Stage	A logical phase; stages run in sequence	Implicit (via `needs:`)	`stages:`
Job	A unit of work; jobs within a stage run in parallel	`jobs.<name>`	Job block
Step	An individual command or reusable action inside a job	`steps:`	`script:`
Runner / Agent	The machine or container that executes a job	`runs-on:`	`tags:`
Artifact	File produced by a job and passed to later jobs	`upload-artifact`	`artifacts:`
Cache	Reused data (dependencies) to speed up later runs	`actions/cache`	`cache:`
Environment	Deployment target (staging, production) with protection rules	`environment:`	`environment:`
Secret	Encrypted credential injected at runtime, never stored in repo	`secrets:`	CI/CD Variables

Fail Fast: Ordering Cheap to Expensive¶

Good pipelines order stages from fastest and cheapest to slowest and most expensive. A lint failure should surface in 30 seconds, not after a 20-minute Docker build. If linting fails, you waste no compute on the expensive stages. This "fail fast" principle minimizes both feedback time and cost. The canonical order for a Python service: lint/format check (seconds) → type check (seconds) → unit tests (one to two minutes) → integration tests (a few minutes) → build artifact/image (two to five minutes) → security scan (one to two minutes) → deploy to staging → E2E tests (tens of minutes) → approve → deploy to production.

Artifacts vs. Caches

Artifacts are outputs you intentionally pass between jobs: a compiled binary, a Docker image digest, a test coverage report, a signed SBOM. They represent the "work product" of a stage and are often retained for audit purposes. Caches are reused inputs that speed up repeated work: the .venv directory, Maven's local repository, the npm node_modules. Caches are an optimization — the pipeline would produce the same result without them, just slower. The distinction matters when debugging: a stale cache can cause mysterious failures; a corrupt artifact is a real problem.

Secrets Management¶

Secrets (database passwords, API keys, deploy credentials) are encrypted values stored in the CI/CD platform's secrets store and injected as environment variables at runtime. They are never stored in the repository — not even in an encrypted form that is checked in. The pipeline references them by name (${{ secrets.DATABASE_URL }}); the platform substitutes the value when the job runs, masking it from logs. Modern platforms support environment-scoped secrets (different database URL for staging vs production), secret rotation, and OIDC-based keyless credentials that replace static secrets entirely for cloud deployments.

pipeline-concepts.yml (generic)

# Illustrates: trigger, job dependency, artifact upload/download, secret, env

on:
  push:
    branches: [main]
  pull_request:               # trigger on any PR

jobs:
  # ── Stage 1: fast verification ─────────────────────────────────
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv run ruff check . && uv run ruff format --check .
      - run: uv run mypy src/

  # ── Stage 2: tests (depends on verify passing) ─────────────────
  test:
    needs: verify                  # explicit dependency — run after verify
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv run pytest --cov=src --cov-report=xml
      - uses: actions/upload-artifact@v4   # upload coverage as artifact
        with:
          name: coverage-report
          path: coverage.xml

  # ── Stage 3: build image (depends on tests passing) ────────────
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and tag
        run: docker build -t myapp:${{ github.sha }} .
      - uses: actions/download-artifact@v4   # download coverage from test job
        with:
          name: coverage-report

  # ── Stage 4: deploy staging (only on main branch, needs build) ──
  deploy-staging:
    needs: build
    runs-on: ubuntu-latest
    environment: staging           # protection rules apply; secrets scoped here
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Deploy
        env:
          DB_URL: ${{ secrets.STAGING_DB_URL }}   # injected at runtime, masked
        run: |
          echo "Deploying ${{ github.sha }} to staging"
          # deploy commands here

Never Log Secrets or Commit Them

CI platforms mask secret values in logs, but if you base64-encode a secret and print the result, the mask doesn't apply. Common accident: echoing an environment variable that contains a secret. Most platforms also prevent secrets from being read in PR workflows from forked repositories — understand this limitation so you don't design pipelines that assume secrets are available in fork PRs. For Git history: if a secret was ever committed, rotate it immediately — even if you deleted it in a subsequent commit. Deleted data is still in the Git history and can be extracted trivially.

Consulting Lens: Reading Any Pipeline

You will walk into clients running GitHub Actions, GitLab CI, Jenkins Declarative Pipelines, Azure Pipelines, or CircleCI. The vocabulary transfers. When you see a new pipeline config, ask: What triggers it? What are the stage dependencies? Where is the fail-fast gate? How are secrets injected? What artifacts are produced and where do they go? How is production deployment gated? Answering these five questions gives you a complete picture of the pipeline's safety properties, and the gaps are where you add value.

Knowledge check

InterviewWhy do well-designed pipelines run linting and unit tests before building Docker images and running E2E tests?

Linting is required by law before Docker builds.
To fail fast — cheap, fast checks catch most problems in seconds, avoiding wasted time and compute running slow, expensive stages on code that was already broken.
E2E tests can only run after linting completes.
Docker builds require linting to complete first or they fail.

Answer

To fail fast — cheap, fast checks catch most problems in seconds, avoiding wasted time and compute running slow, expensive stages on code that was already broken.

Ordering stages cheap-to-expensive gives the fastest feedback and conserves compute resources. A lint failure surfaces in 30 seconds, preventing spending 5–10 minutes building an image and 30 minutes running E2E tests against code that was already broken at the most basic level. This fail-fast structure is a core pipeline design principle that simultaneously improves developer experience (fast feedback) and reduces CI costs.

Knowledge check

ConceptWhat is the difference between a pipeline artifact and a pipeline cache?

They are identical; the terms are interchangeable in CI/CD.
An artifact is an intentional output of a stage passed to later stages (a binary, a test report, a signed image); a cache is reused input that speeds up repeated work (installed dependencies) without affecting the result.
Caches are permanent; artifacts are deleted after each run.
Artifacts are only used for Docker images; caches are for everything else.

Answer

An artifact is an intentional output of a stage passed to later stages (a binary, a test report, a signed image); a cache is reused input that speeds up repeated work (installed dependencies) without affecting the result.

The conceptual distinction: an artifact represents deliberate data flow between pipeline stages — "stage A produces this file; stage B needs it." If the artifact is missing, the pipeline is broken. A cache is a speed optimization — the pipeline produces the same result without it, just more slowly. Stale caches can cause confusing failures; artifacts are auditable, often retained, and may be subject to supply-chain security controls (SBOMs, signatures).

Exercise

Design (in prose or pseudocode) a complete pipeline for a Python FastAPI service that: (1) triggers on every push and PR; (2) runs lint/type check, then tests, then builds a Docker image, then deploys to staging on main branch only; (3) requires a manual approval before deploying to production; (4) injects database credentials from secrets. List each stage, its trigger condition, its dependencies, and one key artifact it produces.

Show solution

Trigger: push (all branches) + pull_request

Stage 1: verify (all branches)
  - Depends on: trigger
  - Steps: ruff check, ruff format --check, mypy
  - Artifact: none (pass/fail gate only)

Stage 2: test (all branches)
  - Depends on: verify
  - Steps: pytest --cov, upload coverage XML
  - Artifact: coverage-report.xml

Stage 3: build (all branches, but push only for image)
  - Depends on: test
  - Steps: docker build, docker push to registry
  - Artifact: image tag (e.g. sha-abc1234)

Stage 4: deploy-staging (main branch only)
  - Depends on: build
  - Environment: staging (secrets: STAGING_DB_URL)
  - Steps: pull image from registry, deploy to staging K8s/ECS
  - Artifact: staging deployment confirmation

Stage 5: e2e-tests (main branch only, after staging deploy)
  - Depends on: deploy-staging
  - Steps: run E2E/smoke tests against staging URL
  - Artifact: E2E test report

Stage 6: approval-gate (main branch only)
  - Depends on: e2e-tests
  - Type: manual approval (GitHub Environment protection rule)

Stage 7: deploy-production (main branch, after approval)
  - Depends on: approval-gate
  - Environment: production (secrets: PROD_DB_URL)
  - Steps: deploy approved image tag to production
  - Artifact: production deployment confirmation + release tag

Key Takeaways

A pipeline = trigger → stages → jobs → steps, executed on runners. The vocabulary transfers across all CI/CD tools.
Artifacts are intentional data flow between stages; caches are speed optimizations.
Environments gate deployment with protection rules and scoped secrets.
Secrets are encrypted, injected at runtime, and must never be committed to the repo.
Order stages fast-and-cheap → slow-and-expensive: fail fast, minimize wasted compute.