How to Set Up Custom Controllers in Kubernetes

Kubernetes ships with a powerful set of built-in controllers — the Deployment controller, the ReplicaSet controller, the Job controller, and dozens more running quietly inside kube-controller-manager. But sooner or later, if you work with Kubernetes long enough, you’ll hit a wall: you need logic that Kubernetes itself doesn’t know how to express. Maybe you want to automatically provision a database whenever someone creates a Database object, or you want to enforce a custom scaling policy based on a metric that isn’t CPU or memory. That’s where custom controllers come in.

In this guide, I’ll walk through what a custom controller actually is, how it fits into Kubernetes’ architecture, and how to build and deploy one from scratch — from the basic reconciliation loop all the way to production-grade concerns like leader election, RBAC, and observability.

What Is a Controller, Really?

At its core, Kubernetes is a declarative system built around a single idea: desired state versus actual state. You tell the API server what you want (via a manifest), and controllers continuously work to make the actual state match that desired state. This is called a reconciliation loop, and it’s the heartbeat of everything Kubernetes does.

A controller:

1. Watches the API server for changes to a resource (via watch on the API, backed by etcd).
2. Compares the observed state to the desired state.
3. Acts to reconcile the difference — creating, updating, or deleting resources as needed.
4. Repeats, forever, reacting to new events and periodically re-syncing.

Built-in controllers do this for native resources like Pods and Deployments. A custom controller does the exact same thing, but for resources you define yourself — usually a Custom Resource Definition (CRD) — or even for built-in resources, if you want to add your own behavior on top of what Kubernetes already does.

Kubernetes Architecture Refresher

Before writing a controller, it helps to understand where it lives in the broader architecture:

• API Server: The front door. All reads and writes go through it, and it persists objects into etcd.
• etcd: The cluster’s source of truth, a distributed key-value store.
• Scheduler: Assigns Pods to Nodes.
• kubelet: Runs on every node, actually starts/stops containers.
• Controller Manager: Runs the built-in controllers as control loops.
• Custom Controller: Just another client of the API server. It doesn’t need special privileges to “be” a controller — it authenticates like any other client, watches resources it cares about, and issues API calls to reconcile them.

This last point is important: a custom controller is not a special kind of process. It’s an ordinary program (often a Go binary, though not exclusively) that talks to the Kubernetes API using a client library, typically running as a Deployment inside the cluster it manages — or even outside the cluster, pointed at a kubeconfig.

When You Need a Custom Controller vs. a CRD Alone

A CRD by itself just defines a new object type — it’s schema and storage, nothing more. If you create a CRD for Website and apply a Website object, absolutely nothing happens unless something is watching for it. That “something” is the controller. Together, a CRD plus a controller form what’s commonly called an Operator — a controller with domain-specific knowledge that automates an operational task a human would otherwise do by hand.

Use a custom controller when:

• You’re building an Operator for a stateful application (databases, message queues, certificate authorities).
• You want to enforce custom policies (e.g., automatically injecting sidecars, labeling resources, or denying non-compliant objects).
• You need to bridge Kubernetes to an external system (cloud resources, DNS records, ticketing systems).

Building a Custom Controller: The Concepts

Most production controllers today are built with client-go, Kubernetes’ official Go client library, often through higher-level tooling like controller-runtime (used by the Operator SDK and Kubebuilder). The core building blocks are:

• Informer: Maintains a local, cache-synced copy of the resources you care about, so you’re not hammering the API server with reads.
• Lister: A read-only, cache-backed accessor built on top of the informer.
• Workqueue: A rate-limited queue that decouples “an event happened” from “process the event,” with automatic retries on failure.
• Reconciler: The function containing your actual business logic.

The typical flow looks like this:

API Server --watch--> Informer --enqueue--> Workqueue --dequeue--> Reconcile()

Step 1: Define a Custom Resource Definition

Let’s build a small controller that watches a custom Website resource and creates a Deployment and Service for it automatically.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: websites.example.com
spec:
  group: example.com
  names:
    kind: Website
    listKind: WebsiteList
    plural: websites
    singular: website
    shortNames: ["ws"]
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer
                  minimum: 1
              required: ["image"]
            status:
              type: object
              properties:
                availableReplicas:
                  type: integer
      subresources:
        status: {}

Apply it:

kubectl apply -f website-crd.yaml
kubectl get crd websites.example.com

Step 2: Scaffold the Controller with Kubebuilder

Kubebuilder gives you a working project skeleton in minutes:

kubebuilder init --domain example.com --repo github.com/yourname/website-operator
kubebuilder create api --group web --version v1 --kind Website

This generates a WebsiteReconciler struct with a Reconcile method — this is where your logic goes.

func (r *WebsiteReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var website webv1.Website
    if err := r.Get(ctx, req.NamespacedName, &website); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    deployment := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      website.Name,
            Namespace: website.Namespace,
        },
        Spec: appsv1.DeploymentSpec{
            Replicas: &website.Spec.Replicas,
            Selector: &metav1.LabelSelector{
                MatchLabels: map[string]string{"app": website.Name},
            },
            Template: corev1.PodTemplateSpec{
                ObjectMeta: metav1.ObjectMeta{
                    Labels: map[string]string{"app": website.Name},
                },
                Spec: corev1.PodSpec{
                    Containers: []corev1.Container{{
                        Name:  "web",
                        Image: website.Spec.Image,
                    }},
                },
            },
        },
    }

    if err := ctrl.SetControllerReference(&website, deployment, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }

    if err := r.Create(ctx, deployment); err != nil && !apierrors.IsAlreadyExists(err) {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

Notice the call to SetControllerReference — this sets an owner reference so that if the Website object is deleted, Kubernetes’ built-in garbage collector cleans up the Deployment automatically.

Step 3: RBAC for the Controller

A controller running inside the cluster needs a ServiceAccount and permissions scoped to exactly what it touches — nothing more.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: website-controller-role
rules:
  - apiGroups: ["web.example.com"]
    resources: ["websites", "websites/status"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: website-controller-binding
subjects:
  - kind: ServiceAccount
    name: website-controller
    namespace: website-system
roleRef:
  kind: ClusterRole
  name: website-controller-role
  apiGroup: rbac.authorization.k8s.io

Step 4: Deploy the Controller

Build and push the image, then deploy it as a standard Kubernetes Deployment:

make docker-build docker-push IMG=yourrepo/website-controller:v0.1.0
make deploy IMG=yourrepo/website-controller:v0.1.0
kubectl get pods -n website-system

Test it:

kubectl apply -f - <<EOF
apiVersion: web.example.com/v1
kind: Website
metadata:
  name: demo-site
spec:
  image: nginx:1.27
  replicas: 2
EOF

kubectl get deployment demo-site

You should see a Deployment named demo-site with 2 replicas appear automatically — created entirely by your controller.

Leader Election for High Availability

Running a single replica of your controller is a single point of failure. Run multiple replicas with leader election enabled so only one is actively reconciling at a time, with automatic failover:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    LeaderElection:   true,
    LeaderElectionID: "website-controller-leader",
})

The non-leader replicas sit idle, watching a Lease object, ready to take over instantly if the leader’s pod dies.

Observability and Troubleshooting

• Structured logging: Log reconcile events with the resource’s namespace/name for traceability.
• Metrics: controller-runtime exposes Prometheus metrics by default (workqueue_depth, reconcile_errors_total) on :8080/metrics.
• Events: Emit Kubernetes Events (kubectl describe website demo-site) so operators see what your controller is doing without digging through logs.
• Common mistakes: forgetting to handle NotFound errors after deletion, not setting owner references (leading to orphaned resources), and reconciling too aggressively without rate limiting, which can overwhelm the API server.

Production Best Practices

• Keep reconcile functions idempotent — they may be called repeatedly for the same state.
• Use finalizers if your controller needs to clean up external resources (cloud load balancers, DNS records) before an object is deleted.
• Version your CRD schema carefully; use conversion webhooks if you need to evolve the API.
• Set resource requests/limits on the controller Pod itself — a runaway controller can destabilize a cluster.
• For disaster recovery, remember that CRs are stored in etcd like everything else — back up etcd regularly if your CRDs hold important state.

CI/CD Integration

In a real DevOps pipeline, the controller image is built and tested in CI (unit tests with envtest, which spins up a real API server without a full cluster), then the manifests are deployed via GitOps tools like Argo CD or Flux, so any change to the CRD or controller Deployment is applied automatically and auditable through Git history.

Summary

Custom controllers are how Kubernetes becomes a true automation platform rather than just a container scheduler. By combining a CRD with a reconciliation loop, you can teach Kubernetes to manage anything — databases, certificates, cloud infrastructure — using the same declarative model it already uses for Pods and Services. Start small, keep your reconcile logic idempotent, lock down RBAC tightly, and add leader election and observability before you trust a controller with production traffic.

References

• Kubernetes Documentation: Custom Resources
• Kubebuilder Book
• client-go on GitHub
• CNCF Operator White Paper
• Kubernetes controller-runtime