How to Set Up Jenkins for Canary Releases

The first time a bad release slipped past my tests and hit 100% of production traffic, I spent the next hour rolling it back while users complained. Canary releases exist exactly to prevent that scenario — you ship the new version to a small slice of traffic first, watch the metrics, and only roll it out further if things look healthy. This guide covers how to implement canary releases using Jenkins as the orchestrator, with Kubernetes as the primary target platform (though the same principles apply elsewhere).

What Is a Canary Release

A canary release gradually shifts traffic from the old version of an application to the new one, typically in stages: 5%, then 25%, then 50%, then 100%. At each stage, you check error rates, latency, and other health signals before proceeding. If something looks wrong, you roll back with minimal user impact, since only a small fraction of traffic ever saw the broken version.

Jenkins Architecture for Canary Deployments

• Controller: Orchestrates the multi-stage rollout, including timed pauses and metric checks between traffic shifts.
• Agent: Executes deployment commands (kubectl, Helm, cloud CLI) and metric queries.
• Pipeline stages: Deploy canary → shift small traffic % → analyze metrics → shift more traffic → repeat → full rollout or rollback.

The key architectural decision is where the actual traffic-splitting happens — options include Kubernetes with a service mesh (Istio, Linkerd), an ingress controller with weighted routing (NGINX Ingress, Traefik), or a cloud load balancer’s native weighted target groups. Jenkins doesn’t do the traffic splitting itself; it orchestrates and gates the process.

Prerequisites

• A Kubernetes cluster with kubectl access from the Jenkins agent
• A traffic-splitting mechanism already in place — this guide uses NGINX Ingress canary annotations for simplicity, but Istio VirtualServices work the same way conceptually
• A metrics source Jenkins can query — Prometheus is used here
• Jenkins (2.4+ LTS) with the Kubernetes CLI and curl/jq available on the agent

Step 1: Install Required Tools on the Jenkins Agent

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
sudo apt install -y jq

Configure kubectl with a service account token scoped to just the namespaces Jenkins needs to deploy to, stored in Jenkins Credentials as a “Secret file” (kubeconfig).

Step 2: Set Up NGINX Ingress Canary Annotations

Define two Kubernetes Deployments — myapp-stable and myapp-canary — and two matching Services, then two Ingress resources:

# ingress-stable.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-stable
spec:
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-stable
            port:
              number: 80
---
# ingress-canary.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "5"
spec:
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: myapp-canary
            port:
              number: 80

Jenkins will programmatically update the canary-weight annotation as the rollout progresses.

Step 3: Write the Canary Pipeline

pipeline {
    agent any

    environment {
        NAMESPACE = 'production'
        IMAGE = "myrepo/myapp:${env.BUILD_NUMBER}"
    }

    stages {
        stage('Checkout') {
            steps {
                git branch: 'main', url: 'https://github.com/yourorg/myapp.git'
            }
        }

        stage('Build and Push Image') {
            steps {
                sh '''
                    docker build -t $IMAGE .
                    docker push $IMAGE
                '''
            }
        }

        stage('Deploy Canary') {
            steps {
                sh '''
                    kubectl set image deployment/myapp-canary myapp=$IMAGE -n $NAMESPACE
                    kubectl rollout status deployment/myapp-canary -n $NAMESPACE --timeout=120s
                '''
            }
        }

        stage('Shift 5% Traffic') {
            steps {
                sh "kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight=5 -n $NAMESPACE --overwrite"
            }
        }

        stage('Analyze at 5%') {
            steps {
                script {
                    checkErrorRate()
                }
            }
        }

        stage('Shift 25% Traffic') {
            steps {
                sh "kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight=25 -n $NAMESPACE --overwrite"
            }
        }

        stage('Analyze at 25%') {
            steps {
                script {
                    checkErrorRate()
                }
            }
        }

        stage('Shift 100% Traffic') {
            steps {
                input message: 'Metrics look healthy — promote canary to 100%?'
                sh '''
                    kubectl set image deployment/myapp-stable myapp=$IMAGE -n $NAMESPACE
                    kubectl rollout status deployment/myapp-stable -n $NAMESPACE --timeout=120s
                    kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight=0 -n $NAMESPACE --overwrite
                '''
            }
        }
    }

    post {
        failure {
            sh "kubectl annotate ingress myapp-canary nginx.ingress.kubernetes.io/canary-weight=0 -n $NAMESPACE --overwrite || true"
            echo 'Canary rollout failed — traffic reverted to stable.'
        }
    }
}

Step 4: Automated Metric Analysis Function

Wait between traffic shifts and query Prometheus to compare canary error rate against the stable baseline:

def checkErrorRate() {
    sleep(time: 3, unit: 'MINUTES')
    def canaryErrors = sh(
        script: '''
            curl -s "http://prometheus:9090/api/v1/query" \
              --data-urlencode 'query=sum(rate(http_requests_total{job="myapp-canary",status=~"5.."}[3m])) / sum(rate(http_requests_total{job="myapp-canary"}[3m]))' \
              | jq -r '.data.result[0].value[1] // "0"'
        ''',
        returnStdout: true
    ).trim().toFloat()

    if (canaryErrors > 0.02) {
        error "Canary error rate ${canaryErrors} exceeds 2% threshold — aborting rollout."
    }
    echo "Canary error rate acceptable: ${canaryErrors}"
}

This is the heart of the canary strategy: Jenkins doesn’t just wait a fixed time and hope for the best, it actively checks a real metric and fails the pipeline (triggering automatic rollback in the post { failure } block) if the canary is unhealthy.

Canary Releases with a Service Mesh

If you’re running Istio instead of NGINX Ingress, the traffic-splitting step changes to updating a VirtualService weight instead of an ingress annotation:

stage('Shift Traffic via Istio') {
    steps {
        sh '''
            kubectl patch virtualservice myapp -n $NAMESPACE --type merge -p \
              '{"spec":{"http":[{"route":[{"destination":{"host":"myapp","subset":"stable"},"weight":75},{"destination":{"host":"myapp","subset":"canary"},"weight":25}]}]}}'
        '''
    }
}

The rest of the pipeline structure — deploy, shift, analyze, repeat — stays identical regardless of which traffic-splitting mechanism you use underneath.

Real-World Workflow

1. A release candidate is built and pushed to the registry.
2. The canary deployment receives the new image; Jenkins verifies the rollout succeeded before touching traffic.
3. Traffic shifts to 5%, Jenkins waits and queries Prometheus; if error rate is healthy, it proceeds automatically.
4. Traffic shifts to 25%, same check repeats.
5. A human approves the final promotion to 100%, at which point the stable deployment is updated and canary weight returns to zero.
6. If any analysis stage fails, the pipeline automatically zeroes canary traffic and the post { failure } block handles cleanup — no manual rollback needed.

Best Practices

• Always start canary weight small (1-5%) — the goal is limiting blast radius, not speed.
• Automate the health check, don’t rely on a human staring at a dashboard for three minutes.
• Pick metrics that reflect user impact directly — error rate and p99 latency, not just CPU usage.
• Keep canary and stable on identical infrastructure (same node pool, same resource limits) so metric comparisons are apples-to-apples.
• Log every traffic-weight change Jenkins makes, for audit and postmortem purposes.

Troubleshooting

• Canary weight change has no effect: Confirm the Ingress controller actually supports canary annotations (this is NGINX-Ingress-specific syntax; other ingress controllers differ).
• Prometheus query returns null: Usually means no traffic hit the canary yet — increase the sleep/wait time or confirm the job label matches your actual metric labels.
• Rollback doesn’t fully restore stable behavior: Make sure the post { failure } block resets weight to zero AND that the stable deployment wasn’t accidentally touched during the failed run.

Automating Rollback Decisions with a Scoring System

A single metric threshold works for simple cases, but production incidents are rarely that clean. A more robust pattern scores several signals together before deciding whether to proceed:

def evaluateCanaryHealth() {
    def errorRate = queryPrometheus('error_rate')
    def p99Latency = queryPrometheus('p99_latency_ms')
    def cpuUsage = queryPrometheus('cpu_usage_pct')

    def issues = []
    if (errorRate > 0.02) issues << "error rate ${errorRate}"
    if (p99Latency > 800) issues << "p99 latency ${p99Latency}ms"
    if (cpuUsage > 85) issues << "CPU usage ${cpuUsage}%"

    if (issues.size() > 0) {
        error "Canary unhealthy — ${issues.join(', ')}. Aborting rollout."
    }
    echo "Canary healthy across all signals."
}

def queryPrometheus(String metric) {
    def queries = [
        error_rate: 'sum(rate(http_requests_total{job="myapp-canary",status=~"5.."}[3m])) / sum(rate(http_requests_total{job="myapp-canary"}[3m]))',
        p99_latency_ms: 'histogram_quantile(0.99, sum(rate(http_request_duration_ms_bucket{job="myapp-canary"}[3m])) by (le))',
        cpu_usage_pct: 'avg(rate(container_cpu_usage_seconds_total{pod=~"myapp-canary.*"}[3m])) * 100'
    ]
    return sh(
        script: "curl -s 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=${queries[metric]}' | jq -r '.data.result[0].value[1] // \"0\"'",
        returnStdout: true
    ).trim().toFloat()
}

This kind of multi-signal check catches problems a single error-rate threshold would miss — a canary that isn’t throwing errors but is quietly burning through CPU or degrading response times is just as much a rollback candidate.

Canary Releases for Database-Backed Changes

Canary deployments get more complicated when a release includes a database schema change, since both the canary and stable versions run against the same database simultaneously. The safe pattern is to make schema changes backward-compatible for at least one release cycle — add new columns without dropping old ones, deploy the canary, confirm health, promote fully, and only then run a follow-up migration that removes anything no longer needed. Trying to run a breaking migration alongside a canary rollout is one of the most common ways canary deployments fail in practice.

FAQs

How is a canary release different from a blue-green deployment? Blue-green switches all traffic at once between two environments; canary shifts traffic gradually and uses live metrics to decide whether to continue — better for catching issues before they affect everyone.

Can I automate the final 100% promotion instead of requiring manual approval? Yes — remove the input step and add one more checkErrorRate() call before the final kubectl set image, but many teams keep a human gate for the last step as a safety net.

What if I don’t use Kubernetes — can I still do canary releases with Jenkins? Yes, the same pattern applies to weighted target groups in AWS ALB, Azure Traffic Manager, or GCP’s traffic splitting for Cloud Run — only the traffic-shifting commands change.

How long should each canary stage run before proceeding? It depends on your traffic volume — enough time to gather a statistically meaningful sample. Low-traffic services may need 10+ minutes per stage; high-traffic services can often move faster.

Summary

Canary releases turn deployments from an all-or-nothing gamble into a controlled, metric-driven rollout. Jenkins is well suited to orchestrate this because it can combine deployment commands, timed waits, live metric queries, and manual approval gates in a single auditable pipeline — automatically rolling back the moment the data says something’s wrong.

References

• Jenkins Pipeline Syntax Reference
• Kubernetes Documentation
• NGINX Ingress Canary Annotations
• Istio Traffic Management Documentation
• Prometheus Query API Documentation