Hybrid Infrastructure: Bridging Both Worlds¶

Estimated time: 20 min read · Part: CI/CD Across Cloud, On-Prem & Hybrid

Hybrid Is Deliberate Design, Not Accident¶

When enterprises talk about “hybrid cloud,” they often mean two different things. The first is an accident: legacy applications stuck on-prem that nobody has migrated yet, alongside new applications built in the cloud, with no intentional connection between them — two separate environments managed as silos. This is not a hybrid architecture; it is a transitional state.

The second meaning is a deliberate architectural pattern: regulated or data-heavy workloads and their compute are intentionally placed on-premises, stateless public-facing or bursty workloads run in the cloud, and the two environments are connected by a private network bridge and managed by a unified control plane. The pipeline deploys to both as a single system. The split is governed by data gravity and compliance placement rules, not by historical accident.

This is the hybrid architecture worth designing for, and the one that enterprises with serious compliance requirements will increasingly converge on as they mature their cloud adoption.

The Placement Decision Framework¶

The central question in hybrid architecture is: for each workload or data set, where should it run? A framework with two primary signals:

1. Where must the data live? If data must stay in a jurisdiction or on-premises by legal requirement, or if it is too large to move economically, compute that operates on that data must also live there. This is data gravity as a placement rule. A 3-petabyte genomic dataset, a real-time market feed, a core banking ledger that regulators require to be on-prem — these anchor their compute.

2. What are the workload’s characteristics? Stateless workloads (API gateways, web front-ends, batch analytics jobs triggered by events, ML inference endpoints without strict data residency requirements) are ideal cloud candidates: they scale horizontally, they tolerate the occasional cold start, and they benefit from cloud auto-scaling during traffic spikes without burning perpetual on-prem capacity.

The result is a layered split: regulated core (on-prem) + elastic consumer tier (cloud) + private network bridge connecting them.

Private Network Bridges¶

Once you know both environments need to exist, they must be connected so the cloud tier can reach on-prem data stores and services without traversing the public internet.

Site-to-site VPN is the simplest option. An IPsec tunnel is established between an on-prem VPN gateway and a cloud VPN gateway (AWS VPN, GCP Cloud VPN, Azure VPN Gateway). Traffic is encrypted in transit; no dedicated physical link is needed. The downside is bandwidth and latency variability: the tunnel goes over the public internet, so peak-hour congestion can affect throughput and latency. Suitable for moderate-bandwidth connectivity (<1 Gbps effectively) and non-latency-critical workloads.

Dedicated interconnects (AWS Direct Connect, Azure ExpressRoute, GCP Cloud Interconnect) use private fiber circuits from the enterprise’s data center to the cloud provider’s edge points of presence. The connection bypasses the public internet entirely. Latency is sub-millisecond and highly consistent; bandwidth can reach 100 Gbps. These circuits are expensive (\(500–\)5,000+/month depending on bandwidth and region) and require lead time to provision. Mandatory for high-throughput data pipelines, financial trading systems, and any workload where latency variability is unacceptable.

PrivateLink / Private Service Connect is a different model: instead of bridging entire networks, it exposes a specific service endpoint from one VPC into another without full network peering. An on-prem PostgreSQL database can be exposed as a private endpoint accessible to cloud workloads without a full site-to-site VPN. Simpler, more surgical, appropriate when the connectivity requirement is service-level rather than network-level.

Consistent Orchestration: Anthos, Arc, and Outposts¶

The operational challenge of hybrid is not the network bridge — it is managing clusters in two different environments without two separate operational teams and two separate toolchains. Fleet management platforms solve this by extending a single control plane across both environments.

Google Anthos (now branded as GKE Enterprise) extends GKE’s management plane to clusters running anywhere: on-prem, AWS, Azure, or bare metal. From the Google Cloud Console, operators manage policies, monitoring, and service mesh (via Anthos Service Mesh, based on Istio) across all clusters in the fleet. Config Sync (a GitOps agent) pushes configuration from a Git repository to every cluster simultaneously. This is the closest thing in the market to a true multi-cloud unified control plane.

Azure Arc extends Azure Resource Manager (ARM) to non-Azure infrastructure: on-prem Kubernetes clusters, AWS EC2 instances, bare metal servers. Once Arc-enabled, these resources appear in the Azure portal and can be managed with Azure Policy, Azure Monitor, Defender for Cloud, and Azure GitOps (Flux). For enterprises with a Microsoft-primary identity strategy (Azure AD), Arc is extremely powerful because it brings on-prem clusters under the same RBAC model as Azure workloads.

AWS Outposts is the inverse approach: instead of managing on-prem via a cloud control plane, AWS ships a physical rack of its own hardware to the customer’s data center. The rack runs the same AWS services (EC2, EKS, ECS, RDS) and appears as a local AWS region. Applications deployed on Outposts use the same AWS APIs, SDKs, and IAM as cloud deployments. The advantage is exact API parity; the disadvantage is cost (Outposts racks are expensive) and the operational overhead of housing AWS hardware on-prem.

Unified Identity Across Environments¶

A hybrid architecture with separate identity systems for cloud and on-prem is operationally painful and a security risk: different credentials for different environments mean different audit logs, different access review processes, and the temptation to create shared service accounts to avoid the friction. The right design is one identity federation across both.

Common patterns: Azure AD (Entra ID) federated with on-prem Active Directory via Azure AD Connect, so the same user principal authenticates to Azure services and on-prem systems. Okta with LDAP synchronization from on-prem Active Directory, federating into AWS IAM Identity Center and GCP via SAML/OIDC. HashiCorp Vault with multiple auth backends (LDAP, Kubernetes, AWS IAM) provides application-level identity federation across environments.

Central Observability: One View of the Whole System¶

The monitoring anti-pattern in hybrid environments is two separate monitoring stacks — one for cloud (CloudWatch, Cloud Monitoring, Azure Monitor) and one for on-prem (Nagios, Zabbix, Prometheus). Operators switch dashboards to debug issues that span the boundary, losing context in the handoff.

The corrective pattern: a central observability backend that ingests telemetry from both environments. Options include Datadog (agent on every host; unified dashboards), Grafana with a remote Prometheus setup (remote_write from on-prem Prometheus to a central Grafana Cloud or self-hosted Grafana), or the OpenTelemetry Collector deployed in both environments shipping to a unified backend (Tempo for traces, Loki for logs, Prometheus for metrics). The OTel Collector is cloud-agnostic and on-prem friendly — it initiates outbound connections, so it works behind firewalls without inbound holes.

Data Synchronization: CDC for Cross-Environment Replication¶

When on-prem databases need to be mirrored to cloud analytics platforms (e.g., a core banking PostgreSQL on-prem replicated to BigQuery for analytics queries), Change Data Capture (CDC) is the standard pattern. Debezium (a Kafka Connect source connector) reads the database’s WAL (write-ahead log) and streams every insert, update, and delete to Kafka. A sink connector (or Dataflow, dbt on BigQuery) consumes the stream and applies changes to the cloud store. Replication lag is typically sub-second. This is how enterprises run analytics in the cloud against operational data that must remain on-prem.

Multi-Cluster Deployment with ArgoCD ApplicationSet¶

Multi-Cluster Context Switching in Shell Scripts¶

#!/usr/bin/env bash
set -euo pipefail

SHA="${1:?Provide image SHA}"
IMAGE="us-central1-docker.pkg.dev/myproject/apps/api:${SHA}"

# Deploy to cloud cluster (GKE)
echo "--- Deploying to cloud cluster ---"
kubectl config use-context gke_myproject_us-central1_prod
kubectl set image deployment/api api="${IMAGE}" --namespace production
kubectl rollout status deployment/api --namespace production --timeout=5m

# Deploy to on-prem cluster (OpenShift via kubeconfig merge)
echo "--- Deploying to on-prem cluster ---"
kubectl config use-context onprem-prod
kubectl set image deployment/api api="${IMAGE}" --namespace production
kubectl rollout status deployment/api --namespace production --timeout=5m

echo "Deployment of ${SHA} complete across both clusters"

Terraform VPN Gateway (Simplified)¶

flowchart TB
    DEV["Developer<br/>git push"] --> CI

    subgraph CLOUD["Public Cloud — Elastic Front-End"]
        CI["CI (GitHub Actions)<br/>Build & Test"] --> REG["Container Registry<br/>(GCR / ECR)"]
        REG --> GKE["Cloud K8s<br/>(GKE / EKS)<br/>stateless, public-facing"]
    end

    subgraph BRIDGE["Private Network Bridge"]
        VPN["VPN / Direct Connect<br/>Encrypted Private Tunnel"]
    end

    subgraph ONPREM["On-Premises — Regulated Core"]
        OKS["On-Prem K8s<br/>(OpenShift / Rancher)"]
        DB[("Regulated<br/>Database<br/>(on-prem)")]
        OKS --- DB
    end

    GKE <-->|"API calls<br/>via private tunnel"| VPN
    VPN <--> OKS

    CD["ArgoCD / Flux<br/>(single GitOps agent<br/>targets both clusters)"] --> GKE
    CD --> OKS

    style CLOUD fill:#1e3a5f,color:#e2e8f0
    style BRIDGE fill:#2a2a1e,color:#e2e8f0
    style ONPREM fill:#2a1e1e,color:#e2e8f0