Offensive Security Researcher & Quantum Cryptography Analyst

How to Run Jenkins on Kubernetes

Abdul Wahab Junaid
How to Run Jenkins on Kubernetes

The first time I moved Jenkins from a lonely EC2 instance to Kubernetes, I was skeptical it would be worth the migration effort. A few months in, I was a convert – dynamic build agents that scale to zero when idle, no more manually provisioning beefy static agents, and a controller that survives node failures without me getting paged at 3 a.m. If you’re tired of babysitting Jenkins infrastructure, running it on Kubernetes is one of the highest-leverage changes you can make. Here’s how I set it up, end to end.

Why Run Jenkins on Kubernetes

A traditional Jenkins setup has a controller plus a fixed pool of static agents that sit around consuming resources even when nothing is building. On Kubernetes, Jenkins can dynamically provision agent pods on demand using the Kubernetes plugin, run the build, and tear the pod down the moment the job finishes. This gets you:

Jenkins Architecture on Kubernetes

At a high level, you have:

Step 1: Prerequisites

Step 2: Install Jenkins Using Helm

helm repo add jenkins https://charts.jenkins.io
helm repo update

kubectl create namespace jenkins

helm install jenkins jenkins/jenkins \
  --namespace jenkins \
  --set controller.serviceType=LoadBalancer \
  --set persistence.enabled=true \
  --set persistence.size=20Gi

This deploys the controller with persistent storage, a default admin user, and the Kubernetes plugin pre-installed. Fetch the initial admin password:

kubectl exec -n jenkins -it svc/jenkins -c jenkins -- \
  /bin/cat /run/secrets/additional/chart-admin-password

Step 3: Set Up RBAC for Dynamic Agents

The Helm chart creates a ServiceAccount and Role automatically, but if you’re doing this manually, here’s the minimum RBAC you need:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-agent
  namespace: jenkins
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "pods/log", "persistentvolumeclaims"]
    verbs: ["create", "delete", "get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins-agent-binding
  namespace: jenkins
subjects:
  - kind: ServiceAccount
    name: jenkins
    namespace: jenkins
roleRef:
  kind: Role
  name: jenkins-agent
  apiGroup: rbac.authorization.k8s.io

Step 4: Configure the Kubernetes Cloud in Jenkins

Go to Manage Jenkins → Clouds → Add a new cloud → Kubernetes. Set:

Click Test Connection to confirm.

Step 5: Define a Pod Template

You can define pod templates in the UI, or better, as code inside your Jenkinsfile using the kubernetes agent directive:

pipeline {
    agent {
        kubernetes {
            yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: maven
    image: maven:3.9-eclipse-temurin-17
    command: ['sleep']
    args: ['infinity']
    resources:
      requests:
        cpu: "500m"
        memory: "512Mi"
      limits:
        cpu: "1"
        memory: "1Gi"
  - name: docker
    image: docker:24-dind
    securityContext:
      privileged: true
"""
        }
    }

    stages {
        stage('Build') {
            steps {
                container('maven') {
                    sh 'mvn -B clean package'
                }
            }
        }
        stage('Build Image') {
            steps {
                container('docker') {
                    sh 'docker build -t myapp:${BUILD_NUMBER} .'
                }
            }
        }
    }
}

Every stage runs inside a specific container within the same pod, and the whole pod disappears once the pipeline finishes.

Step 6: Expose Jenkins Externally

For production, put Jenkins behind an Ingress with TLS:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: jenkins
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts: ["jenkins.mycompany.com"]
      secretName: jenkins-tls
  rules:
    - host: jenkins.mycompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jenkins
                port:
                  number: 8080

Scaling and Resource Management

Set sensible resources.requests and resources.limits on every agent pod template – without them, a runaway build can starve the node and take down other pods. Combine this with a Cluster Autoscaler so the underlying node pool grows and shrinks with build demand instead of you managing node count manually.

Integrating with the Wider Toolchain

Running on Kubernetes makes several other integrations more natural:

Monitoring and Troubleshooting

Security Best Practices

Best Practices

FAQs

Can I run Jenkins itself as a pod without Helm? Yes, raw Deployment/Service/PVC manifests work fine; Helm just saves you from writing them by hand.

Do agent pods need Docker installed on the node? Not with Kaniko or Buildah, which build images without a Docker daemon, making them better suited to Kubernetes-native environments.

How do I handle Jenkins upgrades on Kubernetes? Update the image tag in your Helm values and run helm upgrade; because state lives on a PVC, upgrades are generally low-risk if you snapshot the volume first.

Can I run multiple Jenkins controllers on one cluster? Yes, each in its own namespace with its own PVC – useful for team isolation or blue/green controller upgrades.

Is Kubernetes overkill for a small team? If you’re already running Kubernetes for your applications, it’s usually less overhead than maintaining separate static Jenkins agent VMs. If you have no Kubernetes footprint at all, a simpler Docker-based setup might be more proportionate to your needs.

Summary

Running Jenkins on Kubernetes turns your CI/CD infrastructure from a set of static, hand-maintained machines into an elastic, self-healing system. Install the controller via Helm, configure the Kubernetes cloud plugin, define pod templates as code in your Jenkinsfiles, and let the cluster handle scaling agents up and down automatically. Combined with proper RBAC, resource limits, and Workload Identity for cloud credentials, this setup scales from a five-person team to hundreds of concurrent builds without you manually provisioning a single VM.

References

Exit mobile version