Infrastructure as Code, GitOps & Secrets¶

Estimated time: 22 min read · Part: CI/CD Across Cloud, On-Prem & Hybrid

Why Infrastructure Must Be Code¶

The moment you click “Create VPC” or “Launch instance” in a cloud console and don’t record it, you have created debt. You cannot reproduce that environment: the next time you need it (disaster recovery, a new region, a staging environment for testing), you have to re-click through the console and hope nothing has changed since. You cannot review it: there is no diff, no PR, no audit trail. You cannot audit it: you can’t answer “who changed the firewall rule on Tuesday and why.”

Infrastructure as Code (IaC) is the practice of expressing infrastructure configuration in text files that are stored in version control. IaC is to infrastructure what version control is to application code: it gives you reproducibility, diffability, and auditability as first-class properties.

IaC Tools: The Landscape¶

Terraform / OpenTofu (HCL): the industry-standard IaC tool, used by the majority of platform engineering teams. Terraform uses a declarative language (HCL, HashiCorp Configuration Language) to describe the desired state of infrastructure. terraform plan shows a diff between what Terraform knows (state file) and what the code declares. terraform apply makes the cloud match the declaration. Terraform supports hundreds of providers (AWS, GCP, Azure, Kubernetes, Datadog, GitHub, PagerDuty — nearly everything with an API). OpenTofu is the community fork created after HashiCorp changed Terraform’s license; it is API-compatible and increasingly the choice for teams that require an open-source license.

Pulumi (Python / TypeScript / Go / C#): IaC in real programming languages. Instead of HCL, you write Python classes and functions. This is a significant advantage for teams that are already strong Python engineers: your infrastructure code is typed, testable with pytest, importable as a library, and refactorable with standard IDE tooling. pulumi preview shows a diff; pulumi up applies it. Pulumi supports the same cloud providers as Terraform (many via auto-generated providers from Terraform provider schemas). The tradeoff: steeper learning curve for people who are not engineers, and a smaller ecosystem of community examples.

AWS CDK (Python or TypeScript): the AWS Cloud Development Kit lets you write infrastructure in Python (or TypeScript) and synthesize it to CloudFormation JSON/YAML. AWS-only, but extremely expressive for AWS resources. L2 constructs (higher-level abstractions) let you create a fully configured ECS service with load balancer and security groups in a few lines of code. The generated CloudFormation is managed by the CDK CLI. Well-suited to AWS-centric shops where the team already writes Python.

Bicep: Microsoft’s domain-specific language (DSL) for Azure Resource Manager templates — a significant improvement over writing raw ARM JSON. Azure-only. The right choice when deploying Azure-native services and the team is in the Microsoft ecosystem.

Terraform State Management¶

Terraform tracks what it has created in a state file (terraform.tfstate). By default this is local, which is dangerous for team use: different engineers might have different state files, leading to conflicting applies that create duplicate or orphaned resources. Remote state backends solve this:

Pulumi in Python: Typed Infrastructure¶

GitOps: Git as the Source of Truth¶

GitOps is a deployment model where the Git repository is the single source of truth for the desired state of the system. An in-cluster agent continuously reconciles the live state of the cluster against the declared state in Git. The consequence is a powerful inversion: a deployment is no longer “run a pipeline that pushes to the cluster” — it is “merge a PR, and the cluster converges to match.”

Argo CD is the most widely deployed GitOps tool. It is a Kubernetes controller that watches one or more Git repositories for changes to Kubernetes manifests, Helm charts, or Kustomize overlays. When it detects a diff between Git and the cluster, it applies the Git state. It supports multi-cluster deployments via ApplicationSet, SSO integration, and a web UI that shows the sync status and resource health of every Application.

Flux CD uses a modular approach: the Source Controller watches Git; the Kustomize Controller applies Kustomize overlays; the Helm Controller manages Helm releases; the Image Automation Controller can update the image tag in Git automatically when a new image is pushed. Flux is fully GitOps Toolkit-based: all configuration is expressed as Kubernetes CRDs, which means Flux can manage itself via GitOps.

ArgoCD Application Manifest¶

Secrets Management: Never Commit Plaintext¶

The most expensive mistake in platform engineering is committing a secret to a Git repository. Even if caught and deleted immediately, the secret is in the git history and must be treated as compromised. Rotation is mandatory and urgent. If the repository is public, automated bots scan GitHub continuously for credential patterns (AWS access keys, GCP service account keys, Stripe API keys) and will find the secret within minutes of the push.

Secrets Management Options¶

External Secrets Operator¶

The External Secrets Operator (ESO) runs as a controller in the cluster. It watches ExternalSecret CRD objects, fetches the referenced secret values from an external backend (AWS Secrets Manager, GCP Secret Manager, Vault, etc.), and creates or updates a standard Kubernetes Secret object. The application reads the K8s Secret via envFrom or a volume mount — it never calls the secrets backend directly and doesn’t need credentials to do so.

SOPS: Encrypted Secrets in Git¶

SOPS (Secrets OPerationS, by Mozilla) allows you to commit encrypted secrets to the Git repository. The ciphertext lives in Git alongside the application manifests; only the cluster (or a developer with the decryption key) can read the plaintext. This works particularly well with GitOps: Flux CD has native SOPS decryption support, so the GitOps agent decrypts secrets automatically when applying them to the cluster.

Policy as Code: Admission Controllers¶

GitOps ensures that what is in Git is applied to the cluster. But what prevents someone from submitting a manifest to Git (or directly to kubectl) that violates organizational policy — a Deployment without resource limits, a container running as root, a pod pulling from an unauthorized registry, or an image with no signature?

OPA (Open Policy Agent) with the Gatekeeper integration controller runs as a Kubernetes admission webhook. Every kubectl apply or API server mutation is intercepted; OPA evaluates the request against Rego policies. If the policy fails, the admission is denied. Policies are expressed as Kubernetes CRDs (ConstraintTemplate + Constraint).

Kyverno is a Kubernetes-native policy engine with a YAML-based policy syntax that many teams find more approachable than Rego. It supports both validation (deny non-compliant resources) and mutation (automatically add resource limits if not set, inject sidecar containers, copy secrets across namespaces).

The GitOps Loop Visualized¶

flowchart LR
    subgraph DEVFLOW["Developer Workflow"]
        DEV["Developer<br/>edits manifest<br/>or bumps image tag"] --> PR["Pull Request<br/>(review + approve)"]
        PR --> MERGE["Merge to main"]
    end

    subgraph GIT["Git Repository<br/>(desired state)"]
        MERGE --> REPO["Updated Manifests<br/>/ Helm values<br/>/ Kustomize overlays"]
    end

    subgraph CLUSTER["Kubernetes Cluster"]
        ARGO["ArgoCD / Flux<br/>reconciliation loop<br/>(every 3 min or webhook)"] --> DIFF{"Diff:<br/>Git vs. Live?"}
        DIFF -->|"In sync"| WATCH["Continue watching"]
        DIFF -->|"Out of sync"| APPLY["kubectl apply<br/>(converge to Git)"]
        APPLY --> LIVE["Live cluster state<br/>matches Git"]

        DRIFT["Manual kubectl edit<br/>(drift)"] -.->|"selfHeal=true"| DIFF
    end

    REPO -->|"webhook / poll"| ARGO
    WATCH --> ARGO

    style DEVFLOW fill:#1e3a5f,color:#e2e8f0
    style GIT fill:#1a3a2e,color:#e2e8f0
    style CLUSTER fill:#2a1e3a,color:#e2e8f0