How to Implement Custom Schedulers in Kubernetes

The default Kubernetes scheduler is genuinely good at general-purpose bin-packing, but I’ve hit its limits more than once — GPU-aware placement with topology constraints the default scheduler doesn’t understand, batch workloads that need gang scheduling, and cost-optimization logic that wants to prefer spot instances until a threshold is hit. Each time, the answer wasn’t “fight the default scheduler with a pile of affinity rules” — it was writing or deploying a custom scheduler. This article covers how the scheduling system actually works internally and how to build and deploy your own.

How the Default Scheduler Works Internally

kube-scheduler runs a loop, per pod, through two major phases:

1. Filtering (Predicates) — eliminate nodes that can’t run the pod at all (insufficient resources, node taints without matching tolerations, nodeSelector mismatches, port conflicts).
2. Scoring (Priorities) — rank the remaining feasible nodes using weighted plugins (spread pods across zones, prefer nodes with more free resources, respect pod affinity/anti-affinity) and pick the highest-scoring node.

This is implemented as the Scheduling Framework, a plugin-based pipeline with extension points: QueueSort, PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, PreBind, Bind, PostBind.

Unscheduled Pod
      │
      ▼
 QueueSort → PreFilter → Filter → PostFilter
      │
      ▼
 PreScore → Score → Normalize Score
      │
      ▼
 Reserve → Permit → PreBind → Bind → PostBind

Every custom scheduler either (a) runs as a second, independent scheduler binary selected via spec.schedulerName, or (b) is built as a set of plugins compiled into a customized kube-scheduler binary using the Scheduling Framework’s plugin API. Option (b) is the modern, recommended approach; option (a) is simpler and fine for many use cases.

Approach 1: Deploying a Second Scheduler (Out-of-Tree)

The simplest custom scheduler is literally a second instance of kube-scheduler, or an entirely custom binary, running alongside the default one. Pods opt in via schedulerName.

Deployment for a second scheduler instance:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-custom-scheduler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      component: my-custom-scheduler
  template:
    metadata:
      labels:
        component: my-custom-scheduler
    spec:
      serviceAccountName: my-scheduler-sa
      containers:
        - name: scheduler
          image: registry.k8s.io/kube-scheduler:v1.30.0
          command:
            - kube-scheduler
            - --config=/etc/kubernetes/my-scheduler-config.yaml
          volumeMounts:
            - name: config
              mountPath: /etc/kubernetes
      volumes:
        - name: config
          configMap:
            name: my-scheduler-config

Config referencing a distinct scheduler name so it only picks up pods that request it:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-scheduler-config
  namespace: kube-system
data:
  my-scheduler-config.yaml: |
    apiVersion: kubescheduler.config.k8s.io/v1
    kind: KubeSchedulerConfiguration
    profiles:
      - schedulerName: my-custom-scheduler
    leaderElection:
      leaderElect: true
      resourceNamespace: kube-system
      resourceName: my-custom-scheduler

RBAC (the scheduler needs broad read on pods/nodes and write on bindings/events):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-scheduler-sa
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: my-scheduler-binding
subjects:
  - kind: ServiceAccount
    name: my-scheduler-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:kube-scheduler
  apiGroup: rbac.authorization.k8s.io

A pod opting into this scheduler:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-batch-job
  namespace: production
spec:
  schedulerName: my-custom-scheduler
  containers:
    - name: worker
      image: myregistry/gpu-worker:2.1.0
      resources:
        limits:
          nvidia.com/gpu: 1

kubectl apply -f gpu-batch-job.yaml
kubectl get pod gpu-batch-job -o jsonpath='{.spec.schedulerName}'
kubectl get events --field-selector involvedObject.name=gpu-batch-job

Approach 2: Writing Scheduler Framework Plugins (In-Tree Extension)

For real custom logic, you implement Go interfaces matching extension points and compile them into a custom scheduler binary using k8s.io/kubernetes/cmd/kube-scheduler/app.

A minimal Score plugin skeleton that prefers nodes with a specific label (e.g., preferring spot instances up to a ratio):

package plugins

import (
	"context"
	v1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/runtime"
	framework "k8s.io/kubernetes/pkg/scheduler/framework"
)

type SpotPreference struct {
	handle framework.Handle
}

var _ framework.ScorePlugin = &SpotPreference{}

func (sp *SpotPreference) Name() string {
	return "SpotPreference"
}

func (sp *SpotPreference) Score(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
	nodeInfo, err := sp.handle.SnapshotSharedLister().NodeInfos().Get(nodeName)
	if err != nil {
		return 0, framework.NewStatus(framework.Error, err.Error())
	}
	if nodeInfo.Node().Labels["node-lifecycle"] == "spot" {
		return 100, framework.NewStatus(framework.Success)
	}
	return 10, framework.NewStatus(framework.Success)
}

func (sp *SpotPreference) ScoreExtensions() framework.ScoreExtensions {
	return nil
}

func New(_ runtime.Object, h framework.Handle) (framework.Plugin, error) {
	return &SpotPreference{handle: h}, nil
}

Register it in a custom cmd/scheduler/main.go:

package main

import (
	"os"
	"k8s.io/component-base/cli"
	"k8s.io/kubernetes/cmd/kube-scheduler/app"
	"myorg/scheduler-plugins/plugins"
)

func main() {
	command := app.NewSchedulerCommand(
		app.WithPlugin(plugins.New),
	)
	code := cli.Run(command)
	os.Exit(code)
}

Build and containerize:

go build -o my-scheduler ./cmd/scheduler
docker build -t myregistry/my-scheduler:1.0.0 .
docker push myregistry/my-scheduler:1.0.0

Reference the plugin in the scheduler config so it’s actually enabled:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: spot-aware-scheduler
    plugins:
      score:
        enabled:
          - name: SpotPreference

Real-World Use Cases

• Gang scheduling for ML/batch training jobs, where all-or-nothing pod placement matters (Volcano and Kueue solve this well as off-the-shelf options before you build your own).
• GPU topology-aware scheduling to avoid fragmenting NVLink groups across nodes.
• Cost-aware scheduling preferring spot/preemptible nodes until a defined percentage of workload is on-demand for resilience.
• Data-locality scheduling placing pods near the node holding relevant cached data or a specific storage volume.

Before writing your own scheduler, it’s worth checking whether an existing CNCF project (Volcano, Kueue, YuniKorn) already solves your case — reinventing gang scheduling from scratch is a lot of undifferentiated engineering.

Monitoring Custom Schedulers

Scheduler binaries expose Prometheus metrics on /metrics by default (port 10259 for the default scheduler config pattern). Key metrics to track:

scheduler_scheduling_attempt_duration_seconds
scheduler_pending_pods
scheduler_schedule_attempts_total

A quick PromQL check for scheduling failures on your custom scheduler:

increase(scheduler_schedule_attempts_total{result="error"}[5m])

Extender-Based Scheduling (No Custom Binary Required)

Between “run a second whole scheduler” and “compile custom Go plugins,” there’s a middle path worth knowing: scheduler extenders. An extender is an HTTP webhook the default scheduler calls during filtering/scoring, letting you inject custom logic without building or maintaining a custom binary at all.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
extenders:
  - urlPrefix: "http://scheduler-extender.kube-system.svc:8888"
    filterVerb: "filter"
    prioritizeVerb: "prioritize"
    weight: 5
    enableHTTPS: false
    nodeCacheCapable: false

The extender service just needs to implement the expected HTTP contract, receiving a list of candidate nodes and pod spec, returning filtered/scored results:

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/filter', methods=['POST'])
def filter_nodes():
    data = request.json
    nodes = data['Nodes']['items']
    pod = data['Pod']

    # Example: reject nodes without a specific label
    feasible = [n for n in nodes if n['metadata']['labels'].get('gpu-type') == 'a100']

    return jsonify({
        'Nodes': {'items': feasible},
        'FailedNodes': {}
    })

@app.route('/prioritize', methods=['POST'])
def prioritize_nodes():
    data = request.json
    nodes = data['Nodes']['items']
    scores = [{'Host': n['metadata']['name'], 'Score': 10} for n in nodes]
    return jsonify(scores)

This approach is lower-risk than compiling a custom scheduler binary since you’re not maintaining a fork of kube-scheduler at all — just a small HTTP service — though it does add network latency to every scheduling decision, which matters at high pod-creation rates.

Comparing the Three Approaches

I generally recommend starting with a second scheduler instance for anything that can be expressed via nodeAffinity/taints/tolerations combined with existing plugins, reaching for an extender when you need logic tied to external systems (a cost API, a capacity planning service), and only writing Framework plugins when you need something genuinely novel with zero added latency — GPU topology awareness being the classic example.

Production Considerations

• Run custom schedulers with leaderElect: true and at least 2 replicas for HA — a scheduler outage stalls all new pod placement for pods targeting it.
• Multiple schedulers running concurrently can race to bind the same pod in edge cases; the API server’s optimistic concurrency control (resourceVersion checks) protects against double-binding, but watch for AlreadyBound errors in logs.
• Give custom schedulers their own PriorityClass handling awareness — don’t assume default preemption behavior carries over automatically.

Testing Custom Schedulers Safely

Before trusting a custom scheduler with production traffic, I validate it in isolation using a dedicated test namespace with synthetic pods covering the specific placement scenarios the scheduler is meant to solve:

kubectl create namespace scheduler-test
for i in $(seq 1 10); do
  kubectl run test-pod-$i --image=registry.k8s.io/pause:3.9 \
    --overrides='{"spec":{"schedulerName":"my-custom-scheduler"}}' \
    -n scheduler-test
done
kubectl get pods -n scheduler-test -o wide

Confirm placement matches expectations (correct nodes, correct spread, correct GPU/topology grouping) before rolling out schedulerName changes to real workloads. I also keep a rollback plan ready — since schedulerName is just a field on the pod spec, reverting a Deployment to use default-scheduler again is a one-line change, but any already-scheduled pods won’t move without a rolling restart.

Common Mistakes

• Forgetting schedulerName on pod specs, which silently falls back to default-scheduler and makes it look like your custom scheduler isn’t working.
• Insufficient RBAC (particularly missing bindings write permission), which causes pods to sit in Pending with cryptic forbidden errors in scheduler logs.
• Not handling node affinity/taint evaluation correctly in custom Filter plugins, silently scheduling pods onto tainted or otherwise unsuitable nodes.

Summary

Custom schedulers range from “run a second kube-scheduler binary with a different name” to “write Go plugins against the Scheduling Framework for genuinely novel placement logic.” Start with the former for quick wins, and only build framework plugins when you need logic the built-in filters and scorers fundamentally can’t express — and check for an existing CNCF project first, since gang scheduling and topology-aware placement are largely solved problems already.

References

• Kubernetes Docs: Scheduler Configuration
• Kubernetes Docs: Scheduling Framework
• Kubernetes Docs: Multiple Schedulers
• CNCF Volcano Project
• CNCF Kueue Project