How to Set Up Horizontal Pod Autoscaling with Custom Metrics in Kubernetes

CPU-based autoscaling is the default everyone starts with, and it’s also frequently the wrong signal. I’ve managed services where CPU utilization stayed flat under load because the actual bottleneck was queue depth or downstream latency — the HPA would sit there doing nothing while a message queue backed up for twenty minutes. Once you move past CPU/memory autoscaling into custom metrics, the HPA becomes something that actually reflects your application’s real load characteristics. This article covers the full pipeline: exposing an application metric, wiring it through Prometheus Adapter, and configuring the HPA to scale on it.

Why Custom Metrics Autoscaling

The built-in autoscaling/v2 HPA natively supports CPU and memory (via the resource metric type) plus three other categories:

• Pods metrics — per-pod custom metrics (e.g., requests-per-second per pod), averaged across all pods of the target
• Object metrics — a metric describing a Kubernetes object itself (e.g., queue depth on an external message broker, or Ingress request count)
• External metrics — metrics from a source entirely outside Kubernetes (e.g., an SQS queue depth, a cloud load balancer’s request count)

Custom and external metrics both require the Custom Metrics API or External Metrics API to be registered with the cluster, which is what Prometheus Adapter (or a cloud-specific adapter) provides.

Architecture

Application → exposes /metrics (Prometheus format)
      │
      ▼
Prometheus → scrapes and stores the metric
      │
      ▼
Prometheus Adapter → exposes metric via custom.metrics.k8s.io API
      │
      ▼
HorizontalPodAutoscaler → queries the API, adjusts replicas
      │
      ▼
Deployment → scaled

Step 1: Instrument Your Application

Your application needs to expose a Prometheus-format metric reflecting real load. A Node.js example using prom-client, tracking an in-flight request gauge:

const client = require('prom-client');
const register = new client.Registry();

const inFlightRequests = new client.Gauge({
  name: 'http_requests_in_flight',
  help: 'Number of in-flight HTTP requests',
});
register.registerMetric(inFlightRequests);

app.use((req, res, next) => {
  inFlightRequests.inc();
  res.on('finish', () => inFlightRequests.dec());
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

Ensure the Deployment exposes this for Prometheus scraping:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "3000"
        prometheus.io/path: "/metrics"
    spec:
      containers:
        - name: order-service
          image: myregistry/order-service:2.3.0
          ports:
            - containerPort: 3000

Step 2: Confirm Prometheus Is Scraping It

kubectl port-forward -n monitoring svc/kube-prometheus-kube-prome-prometheus 9090:9090

Query in the Prometheus UI or via API:

curl -s 'http://localhost:9090/api/v1/query?query=http_requests_in_flight' | jq .

Step 3: Install and Configure Prometheus Adapter

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring \
  --set prometheus.url=http://kube-prometheus-kube-prome-prometheus.monitoring.svc \
  --set prometheus.port=9090

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter-config
  namespace: monitoring
data:
  config.yaml: |
    rules:
      - seriesQuery: 'http_requests_in_flight{namespace!="",pod!=""}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
            pod: {resource: "pod"}
        name:
          matches: "^http_requests_in_flight"
          as: "http_requests_in_flight"
        metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

kubectl apply -f prometheus-adapter-config.yaml
kubectl rollout restart deployment prometheus-adapter -n monitoring

Verify the metric is now served through the Kubernetes API:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_in_flight" | jq .

Expected output:

{
  "kind": "MetricValueList",
  "items": [
    {
      "describedObject": {"kind": "Pod", "name": "order-service-7d8f9-abc12", "namespace": "production"},
      "metricName": "http_requests_in_flight",
      "timestamp": "2026-08-02T10:15:00Z",
      "value": "12"
    }
  ]
}

Step 4: Define the HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_in_flight
        target:
          type: AverageValue
          averageValue: "15"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

kubectl apply -f hpa.yaml
kubectl get hpa order-service-hpa -n production -w

NAME                 REFERENCE                   TARGETS   MINPODS   MAXPODS   REPLICAS
order-service-hpa    Deployment/order-service     18/15     3         20        4

The behavior block matters a lot in production: fast, aggressive scale-up (stabilizationWindowSeconds: 0) responds quickly to load spikes, while a slower scale-down (stabilizationWindowSeconds: 300) prevents replica count from flapping down and back up during brief lulls in traffic.

External Metrics: Scaling on a Message Queue

For queue-depth-driven scaling — a very common real-world pattern — use the External Metrics API instead. This example assumes an SQS-compatible adapter or a Prometheus exporter surfacing queue depth:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter-config
  namespace: monitoring
data:
  config.yaml: |
    rules:
      - seriesQuery: 'sqs_queue_depth{queue_name!=""}'
        resources:
          overrides:
            namespace: {resource: "namespace"}
        name:
          matches: "^sqs_queue_depth"
          as: "sqs_queue_depth"
        metricsQuery: 'avg(<<.Series>>{<<.LabelMatchers>>}) by (queue_name)'

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 2
  maxReplicas: 50
  metrics:
    - type: External
      external:
        metric:
          name: sqs_queue_depth
          selector:
            matchLabels:
              queue_name: order-processing
        target:
          type: AverageValue
          averageValue: "100"

This scales queue-worker replicas up as messages pile up in the queue, and back down as the queue drains — a much more direct signal than CPU for a worker whose job is entirely I/O-bound queue consumption.

Combining Multiple Metrics

An HPA can evaluate several metrics simultaneously; it always scales to whichever metric demands the highest replica count:

metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_in_flight
      target:
        type: AverageValue
        averageValue: "15"

This is a sensible defensive pattern — CPU as a backstop in case the custom metric pipeline has an outage, plus the custom metric as the primary, more accurate signal.

Monitoring the HPA Itself

kube_horizontalpodautoscaler_status_current_replicas
kube_horizontalpodautoscaler_status_desired_replicas
kube_horizontalpodautoscaler_spec_target_metric

A PromQL alert for an HPA pinned at maxReplicas (a sign you may need to raise the ceiling or investigate a real capacity problem):

kube_horizontalpodautoscaler_status_current_replicas
  == kube_horizontalpodautoscaler_spec_max_replicas

Production Considerations

• Prometheus Adapter is a single point of failure for custom-metric scaling — run it with multiple replicas and monitor its own health; if it’s down, HPAs relying on custom metrics stop scaling (they don’t crash, but they stop reacting to load changes).
• Combine with Pod Disruption Budgets so aggressive scale-down doesn’t violate availability guarantees during traffic dips (see the companion PDB articles).
• Tune behavior blocks deliberately — default HPA behavior can be too conservative for spiky traffic patterns or too aggressive for cost-sensitive workloads.
• Load test your scaling policy, not just your application — a correctly-configured HPA that reacts too slowly is functionally similar to no autoscaling at all during a real traffic spike.

Interaction with Cluster Autoscaler

Custom-metric HPA scaling only adjusts pod replica count — it doesn’t create nodes. If your cluster doesn’t have spare capacity, new replicas from an aggressive scale-up event just sit Pending until the Cluster Autoscaler (or Karpenter) notices and provisions new nodes, which can take a minute or more depending on cloud provider boot times. This gap matters a lot for genuinely spiky workloads:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-autoscaler-status
  namespace: kube-system

kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml
kubectl get pods -n production -l app=order-service --field-selector status.phase=Pending

If you consistently see Pending pods during scale-up events, consider running a small buffer of over-provisioned low-priority pods (the “pause pod” pattern) that get preempted immediately when real workload pods need the capacity, effectively pre-warming node headroom:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning-buffer
  namespace: kube-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: overprovisioning-buffer
  template:
    metadata:
      labels:
        app: overprovisioning-buffer
    spec:
      priorityClassName: overprovisioning
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: "1"
              memory: 1Gi
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: overprovisioning
value: -1
globalDefault: false

Cost Implications of Aggressive Autoscaling

Custom-metric autoscaling that reacts quickly is great for availability but can be expensive if not bounded carefully — I’ve seen a metric pipeline bug cause replica count to spike from 4 to the maxReplicas ceiling and stay there for hours before anyone noticed, simply because nothing was watching cost alongside performance.

sum(kube_horizontalpodautoscaler_status_current_replicas) by (horizontalpodautoscaler)
  * on(horizontalpodautoscaler) group_left
  kube_pod_container_resource_requests{resource="cpu"}

A rough cost-tracking query like this, visualized alongside your HPA dashboards, at least makes runaway scaling visible quickly rather than showing up only on the next cloud bill.

Troubleshooting Custom Metrics HPA Issues

kubectl describe hpa shows unknown for the custom metric. This is the most common failure mode and traces back to one of three things: Prometheus Adapter isn’t running, the ConfigMap rule doesn’t match your metric’s actual label set, or the metric simply has no recent data points (Prometheus Adapter queries a rolling window and returns nothing if the series is stale).

kubectl logs -n monitoring -l app=prometheus-adapter --tail=50
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[].name'

If your metric name doesn’t appear in that resource list at all, the adapter’s rule config isn’t matching — double check the seriesQuery regex against your actual Prometheus metric name and labels.

HPA scales up correctly but never scales back down. Check the behavior.scaleDown.stabilizationWindowSeconds value — if it’s very high (or you’re hitting the default 300s and expecting faster), that’s working as designed, not a bug. Also verify the metric itself is actually dropping:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_in_flight" | jq '.items[].value'

Replica count oscillates rapidly (flapping). This is almost always an undersized stabilizationWindowSeconds on scale-down combined with a noisy metric — either smooth the underlying metric with a longer Prometheus rate() window, or increase the stabilization window, or both.

Common Mistakes

• Scaling on a metric that doesn’t actually reflect load (classic CPU-only scaling for I/O-bound services) instead of identifying the true bottleneck metric first.
• Not setting maxReplicas conservatively enough, letting a runaway metric (or a bug causing metric spikes) scale a Deployment into a cost incident.
• Forgetting stabilizationWindowSeconds on scale-down, causing replica count to oscillate rapidly under fluctuating load.
• Missing RBAC for Prometheus Adapter to read metrics from Prometheus, causing silent unknown metric errors in kubectl describe hpa output.

Summary

Custom metrics autoscaling closes the gap between “what Kubernetes can measure by default” and “what your application’s real bottleneck actually is.” The pipeline — instrument your app, scrape with Prometheus, expose through Prometheus Adapter, consume via the HPA’s Pods/Object/External metric types — is more setup than CPU-based autoscaling, but it’s the difference between an HPA that reacts to real load and one that just watches a proxy metric that happens to correlate with load some of the time.

References

• Kubernetes Docs: Horizontal Pod Autoscaling
• Kubernetes Docs: HPA Walkthrough
• Prometheus Adapter (GitHub)
• Kubernetes Custom Metrics API (GitHub)