There’s a meaningful difference between “I have Jenkins running builds” and “I have a CI/CD pipeline” — the latter implies a complete, repeatable path from a code commit to a running, verified change in production, with rollback built in at every stage. This article covers that complete pipeline design, tool-agnostically where possible, and with a focus on the GitOps pattern that’s become the dominant approach for Kubernetes specifically.
Two Competing Deployment Models
- Push-based CI/CD: the CI system (Jenkins, GitHub Actions, GitLab CI) builds an image and then directly runs
kubectl applyorhelm upgradeagainst the cluster. - Pull-based GitOps: the CI system only builds and pushes an image, then updates a Git repository containing the desired manifests. A cluster-side operator (Argo CD, Flux) continuously reconciles the cluster to match that Git state.
GitOps has become the more common pattern for Kubernetes-native pipelines because it removes the need to give CI systems direct write credentials to the cluster, and it makes Git itself the audit log and rollback mechanism.
Pipeline Architecture (GitOps Pattern)
Developer → git push → CI (build+test+push image)
│
▼
Update image tag in a
separate "manifests" repo
│
▼
Argo CD / Flux detects the
change and reconciles cluster
│
▼
Cluster converges to
desired state
Stage 1: CI — Build, Test, Push
A GitHub Actions example covering build and test before anything touches Kubernetes:
name: CI
on:
push:
branches: [main]
jobs:
build-test-push:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: |
docker build --target test -t myapp-test .
docker run --rm myapp-test
- name: Build and push image
run: |
IMAGE_TAG=registry.example.com/myapp:${{ github.sha }}
docker build -t $IMAGE_TAG .
echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login registry.example.com -u ci --password-stdin
docker push $IMAGE_TAG
- name: Update manifests repo
run: |
git clone https://x-access-token:${{ secrets.MANIFESTS_TOKEN }}@github.com/example/manifests.git
cd manifests
sed -i "s|image: registry.example.com/myapp:.*|image: registry.example.com/myapp:${{ github.sha }}|" production/deployment.yaml
git commit -am "Update myapp to ${{ github.sha }}"
git push
Note that CI’s job ends at pushing to the manifests repo — it never touches the cluster directly.
Stage 2: CD — Argo CD Reconciliation
Install Argo CD:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Define an Application pointing at the manifests repo:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: myapp
namespace: argocd
spec:
project: default
source:
repoURL: https://github.com/example/manifests.git
targetRevision: main
path: production
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
kubectl apply -f myapp-application.yaml
Check sync status:
kubectl -n argocd get application myapp
NAME SYNC STATUS HEALTH STATUS
myapp Synced Healthy
selfHeal: true is the key GitOps guarantee — if someone manually kubectl edits something in the cluster that drifts from Git, Argo CD reverts it automatically, keeping Git as the actual single source of truth rather than just documentation of intent.
The Deployment Being Managed
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
namespace: production
spec:
replicas: 4
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: registry.example.com/myapp:latest
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
resources:
requests:
cpu: 250m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
Progressive Delivery: Canary and Blue-Green
For anything riskier than a straightforward rolling update, Argo Rollouts extends Deployments with canary/blue-green strategies:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: myapp
namespace: production
spec:
replicas: 5
strategy:
canary:
steps:
- setWeight: 20
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 100
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: registry.example.com/myapp:latest
This routes 20% of traffic to the new version, waits, checks metrics, then proceeds — automatically pausing or rolling back if error rates spike, when wired to a metrics provider like Prometheus.
RBAC for the Pipeline
Argo CD’s own ServiceAccount needs namespace-scoped permissions to manage the resources it’s responsible for — scope it, don’t hand it cluster-admin by default:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: argocd-application-controller
namespace: production
rules:
- apiGroups: ["apps", ""]
resources: ["deployments", "services", "configmaps", "secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
Rolling Back
With GitOps, rollback is a Git operation, not a cluster operation:
git revert <bad-commit-sha>
git push
Argo CD detects the reverted manifest and reconciles the cluster back automatically. This is a meaningfully different (and generally safer) rollback story than helm rollback or kubectl rollout undo, since the full history lives in Git rather than in cluster-side release metadata.
Monitoring Pipeline Health
kubectl -n argocd get applications
argocd app get myapp
argocd app history myapp
Feed Argo CD’s own metrics into Prometheus for pipeline-level dashboards and alerting on sync failures:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: argocd-metrics
namespace: argocd
spec:
selector:
matchLabels:
app.kubernetes.io/name: argocd-metrics
endpoints:
- port: metrics
High Availability for the Pipeline Itself
Argo CD’s application-controller supports sharding across multiple replicas for large numbers of Applications; the API server and repo-server components scale horizontally as ordinary Deployments. For a genuinely HA pipeline, back Argo CD’s Redis (used for caching) with a replicated configuration rather than a single Pod.
Environment Promotion: Moving a Build Through Stages
A single build artifact should move through dev, staging, and production without being rebuilt at each stage — rebuilding introduces the risk that the image running in production isn’t actually the exact same bytes that were tested in staging. The GitOps manifests repo structure typically reflects this directly:
manifests/
dev/
deployment.yaml
staging/
deployment.yaml
production/
deployment.yaml
A promotion pipeline copies the image tag, not the whole config, from one environment’s manifest to the next:
#!/bin/bash
STAGING_TAG=$(yq '.spec.template.spec.containers[0].image' staging/deployment.yaml | cut -d: -f2)
yq -i ".spec.template.spec.containers[0].image = \"registry.example.com/myapp:${STAGING_TAG}\"" production/deployment.yaml
git commit -am "Promote ${STAGING_TAG} to production"
git push
This can be gated behind a manual approval step in most CI systems, so promotion to production is a deliberate human action even though everything else is automated:
jobs:
promote-to-production:
runs-on: ubuntu-latest
environment:
name: production
steps:
- run: ./promote.sh
The environment: production block in GitHub Actions, for instance, can require a specific reviewer’s approval before the job proceeds — combining full automation with an explicit human checkpoint at exactly the one stage where that’s usually still wanted.
Automated Testing Gates Before Deployment
A pipeline that deploys immediately after a successful build, without any deeper verification, is really just automated pushing rather than genuine continuous delivery. A more complete gate sequence typically includes:
jobs:
test:
steps:
- name: Unit tests
run: go test ./... -cover
- name: Integration tests against ephemeral cluster
run: |
kind create cluster --name ci-test
kubectl apply -f k8s/
kubectl wait --for=condition=available deployment/myapp --timeout=60s
./run-integration-tests.sh
kind delete cluster --name ci-test
- name: Security scan
run: trivy image registry.example.com/myapp:${{ github.sha }}
Using kind (Kubernetes-in-Docker) to spin up a genuinely disposable cluster for integration testing — rather than testing only against mocked dependencies — catches an entire category of bug that unit tests alone miss: misconfigured probes, wrong Service selectors, RBAC that’s too restrictive for the app to actually function.
Push-Based CI/CD Still Has a Place
Despite GitOps’ clear advantages for most production workloads, it’s worth being fair to the push-based model rather than treating it as strictly obsolete. Push-based deploys are simpler to reason about for small teams, don’t require standing up and operating a separate reconciliation tool, and give faster feedback for genuinely fast-moving early-stage projects where the operational overhead of a full GitOps setup isn’t yet worth it. A reasonable middle-ground pattern many teams adopt: push-based deploys to development and staging environments, where speed of iteration matters more than auditability, and GitOps specifically for production, where the stronger guarantees around drift detection and rollback actually earn their keep.
jobs:
deploy-dev:
steps:
- run: kubectl apply -f k8s/dev/ -n development
deploy-production:
steps:
- run: |
git clone $MANIFESTS_REPO
cd manifests && sed -i "s|tag: .*|tag: ${{ github.sha }}|" production/values.yaml
git commit -am "Deploy ${{ github.sha }}" && git push
Measuring Pipeline Health, Not Just Individual Deploys
A mature CI/CD setup tracks metrics about the pipeline itself, not just whether the last deploy succeeded — the four DORA metrics (deployment frequency, lead time for changes, change failure rate, and time to restore service) are the widely referenced framework for this:
# Deployment frequency, from Argo CD application history
argocd app history myapp | wc -l
# Change failure rate, approximated from rollback frequency
git log --oneline manifests/production/ | grep -c "revert"
Tracking these over time — even informally — surfaces whether a pipeline is actually improving delivery speed and reliability, or just moving the same manual toil into a slightly more automated shape without changing the underlying failure rate.
Common Mistakes
- Giving CI direct cluster-admin credentials for push-based deploys instead of adopting the pull-based GitOps model, creating a large blast radius if CI credentials leak.
- Skipping
selfHeal, allowing manual cluster edits to silently drift from Git without anyone noticing. - No canary or progressive delivery step for high-risk changes, turning every deploy into an all-or-nothing bet.
- Not setting
prune: truecarefully — combined with a broadpath, it can delete resources unintentionally if the manifests repo structure changes unexpectedly.
Summary
A real Kubernetes CI/CD pipeline separates concerns cleanly: CI builds, tests, and produces artifacts; CD (increasingly via GitOps tools like Argo CD or Flux) reconciles the cluster to match a Git-defined desired state. This split removes the need for CI to hold cluster credentials, makes Git the audit trail and rollback mechanism, and — combined with progressive delivery strategies — turns deployment from a risky manual event into a routine, mostly self-verifying process.
References
- Argo CD documentation: https://argo-cd.readthedocs.io/en/stable/
- Argo Rollouts documentation: https://argoproj.github.io/argo-rollouts/
- Flux documentation: https://fluxcd.io/flux/
- GitOps principles (OpenGitOps, CNCF): https://opengitops.dev/
