The first time a bad deploy took down a production service on my watch, it wasn’t because the code was untested — it was because we shipped it to 100% of traffic at once with no gradual rollout. Canary releases exist precisely to prevent that kind of all-or-nothing exposure. In this guide, I’ll walk through several ways to implement canary releases in Kubernetes, from simple manual approaches using native primitives to full progressive delivery with Argo Rollouts and Flagger on AWS EKS.
What Is a Canary Release?
A canary release is a deployment strategy where a new version of an application is rolled out to a small subset of users or traffic first, monitored for errors or performance regressions, and only promoted to full traffic if it looks healthy. The name comes from “canary in a coal mine” — a small, expendable early warning system. Unlike a blue-green deployment (which switches all traffic at once between two full environments), a canary is about gradual, traffic-weighted exposure.
Canary Approaches in Kubernetes
There are roughly three tiers of sophistication:
- Manual canary using two Deployments and a shared Service — simplest, works with any cluster, but crude traffic splitting
- Service mesh-based canary (Istio, Linkerd, App Mesh) — precise percentage-based traffic splitting at L7
- Progressive delivery controllers (Argo Rollouts, Flagger) — automated, metric-driven canary promotion/rollback
Let’s build up through all three.
Method 1: Manual Canary with Native Kubernetes Objects
The core trick here: a Kubernetes Service selects pods by label, not by Deployment name. If both your stable and canary Deployments share a common label that the Service selects on, traffic gets load-balanced across both — and the ratio of traffic is roughly proportional to the ratio of pod counts.
Stable deployment (90% of traffic via 9 replicas):
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-stable
spec:
replicas: 9
selector:
matchLabels:
app: web
track: stable
template:
metadata:
labels:
app: web
track: stable
spec:
containers:
- name: web
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/web:v1.4.0
ports:
- containerPort: 8080
Canary deployment (10% of traffic via 1 replica):
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-canary
spec:
replicas: 1
selector:
matchLabels:
app: web
track: canary
template:
metadata:
labels:
app: web
track: canary
spec:
containers:
- name: web
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/web:v1.5.0
ports:
- containerPort: 8080
Shared Service (notice it selects only on app: web, matching both):
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web
ports:
- port: 80
targetPort: 8080
kubectl apply -f web-stable.yaml -f web-canary.yaml -f web-service.yaml
kubectl get pods -l app=web -L track
NAME READY STATUS TRACK
web-stable-7d8f9c6b5d-abc12 1/1 Running stable
web-stable-7d8f9c6b5d-def34 1/1 Running stable
...
web-canary-6b9d7f8c4-xyz99 1/1 Running canary
This gives you roughly 10% traffic to canary (1 out of 10 pods), but it’s coarse — real traffic percentages depend on kube-proxy’s load balancing, which isn’t guaranteed to be perfectly even, especially at low replica counts. It’s a fine starting point, but for anything traffic-sensitive you want more precision.
Method 2: Service Mesh-Based Canary with Istio
Istio gives you exact percentage-based traffic splitting independent of pod counts, using VirtualService and DestinationRule.
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: web
spec:
host: web
subsets:
- name: stable
labels:
track: stable
- name: canary
labels:
track: canary
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: web
spec:
hosts:
- web
http:
- route:
- destination:
host: web
subset: stable
weight: 90
- destination:
host: web
subset: canary
weight: 10
Now traffic splitting is exact, regardless of replica counts. To promote the canary, simply adjust weights incrementally — 10 → 25 → 50 → 100 — while watching metrics between each step.
kubectl apply -f web-virtualservice.yaml
kubectl get virtualservice web -o yaml
Method 3: Automated Progressive Delivery with Argo Rollouts
Manually watching dashboards and editing weight percentages doesn’t scale across dozens of services. Argo Rollouts replaces the standard Deployment object with a Rollout CRD that natively understands canary strategies, integrates with metrics providers, and can automatically pause, promote, or roll back.
Install Argo Rollouts:
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
Install the kubectl plugin for convenience:
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
A Rollout with an automated, metric-gated canary strategy:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: web
spec:
replicas: 10
strategy:
canary:
steps:
- setWeight: 10
- pause: { duration: 5m }
- analysis:
templates:
- templateName: success-rate
- setWeight: 25
- pause: { duration: 5m }
- setWeight: 50
- pause: { duration: 5m }
- setWeight: 100
trafficRouting:
istio:
virtualService:
name: web
routes:
- primary
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/web:v1.5.0
ports:
- containerPort: 8080
An AnalysisTemplate that queries Prometheus for error rate and automatically halts promotion if it exceeds a threshold:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
spec:
metrics:
- name: success-rate
interval: 1m
successCondition: result[0] >= 0.95
failureLimit: 3
provider:
prometheus:
address: http://prometheus.monitoring.svc:9090
query: |
sum(rate(http_requests_total{app="web",track="canary",status!~"5.."}[5m]))
/
sum(rate(http_requests_total{app="web",track="canary"}[5m]))
Trigger a new rollout by updating the image, exactly like a Deployment:
kubectl argo rollouts set image web web=123456789012.dkr.ecr.us-east-1.amazonaws.com/web:v1.6.0
kubectl argo rollouts get rollout web --watch
Watch the live promotion in the terminal UI:
kubectl argo rollouts dashboard
If the analysis fails, Argo Rollouts automatically aborts and rolls back to the stable version — no human needed to catch it at 3 a.m.
Method 4: Flagger (Alternative to Argo Rollouts)
Flagger achieves similar automated canary analysis but works as a separate operator watching standard Deployments rather than replacing them with a CRD, which some teams prefer for lower migration friction.
helm repo add flagger https://flagger.app
helm install flagger flagger/flagger \
--namespace istio-system \
--set meshProvider=istio \
--set metricsServer=http://prometheus.istio-system:9090
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
name: web
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: web
service:
port: 80
analysis:
interval: 1m
threshold: 5
stepWeight: 10
maxWeight: 50
metrics:
- name: request-success-rate
thresholdRange:
min: 99
interval: 1m
Canary on AWS EKS Without a Service Mesh
If you’re not running Istio or Linkerd, the AWS Load Balancer Controller with an ALB Ingress can do weighted target-group routing directly, which Argo Rollouts also supports natively:
spec:
strategy:
canary:
steps:
- setWeight: 20
- pause: { duration: 10m }
- setWeight: 100
trafficRouting:
alb:
ingress: web-ingress
servicePort: 80
This avoids needing a full service mesh if ALB-level weighting is sufficient for your use case.
Monitoring During Canary Rollouts
Whatever mechanism you use, you need real signal to judge canary health:
- Error rate — HTTP 5xx ratio, application-level exceptions
- Latency percentiles — p50/p95/p99, since averages hide tail regressions
- Business metrics — checkout completion rate, signup conversion, anything domain-specific that a purely technical metric might miss
kubectl top pods -l track=canary
kubectl logs -l track=canary --tail=100 -f
Common Mistakes
- Sizing the canary too small to be statistically meaningful — 1 pod out of 200 replicas may not see enough traffic to catch a real bug before you’ve already promoted.
- Not automating rollback — a human staring at a dashboard at 2 a.m. is a worse safety net than an automated analysis gate.
- Ignoring stateful side effects — canary pods writing to a shared database with a new (and possibly incompatible) schema can corrupt data even if the canary itself “looks healthy” from a request-rate perspective. Always pair schema changes with backward-compatible migrations.
- Forgetting session affinity implications — if your app relies on sticky sessions, a percentage-based canary can cause a single user to bounce between old and new versions inconsistently.
Best Practices
- Start canaries small (5–10%) and hold for a meaningful observation window before increasing.
- Automate promotion/rollback decisions based on real metrics rather than manual judgment calls under time pressure.
- Always pair canary releases with proper readiness probes so unhealthy canary pods never receive traffic in the first place.
- Version your canary AnalysisTemplates and Rollout manifests in Git — treat the deployment strategy itself as code, reviewed like anything else.
Summary
Canary releases reduce blast radius by exposing new versions to a small slice of traffic before full rollout. Kubernetes’ native label-selector Services give you a crude but zero-dependency starting point; a service mesh like Istio gives you precise weighted routing; and progressive delivery tools like Argo Rollouts or Flagger add automated, metric-driven promotion and rollback on top. On EKS, you can achieve meaningful canary behavior even without a full service mesh by using ALB weighted target groups through the AWS Load Balancer Controller. Choose the tier of sophistication that matches your traffic volume and risk tolerance — but at minimum, never ship a new version to 100% of production traffic in one step.
