How to Set Up Custom Metrics in Kubernetes with Prometheus Adapter

CPU-based autoscaling is a reasonable default until it isn’t. I ran a message queue consumer service for a while that scaled beautifully on CPU right up until the queue backed up during a traffic spike — CPU usage stayed flat because the bottleneck wasn’t compute, it was queue depth. The HPA had no idea anything was wrong. That’s when I actually needed custom metrics, and Prometheus Adapter is the standard way to get there.

The Problem With CPU/Memory-Only Autoscaling

The Horizontal Pod Autoscaler (HPA) natively supports CPU and memory via the metrics.k8s.io API (served by metrics-server). That’s fine for compute-bound workloads, but plenty of real-world scaling signals live elsewhere: queue depth, requests-per-second, active connections, custom business metrics like “pending orders.” None of that is visible to the HPA out of the box.

How Prometheus Adapter Fits In

Kubernetes defines several metrics APIs:

• metrics.k8s.io — resource metrics (CPU/memory), served by metrics-server.
• custom.metrics.k8s.io — arbitrary metrics tied to Kubernetes objects (pods, services, etc.), like http_requests_per_second for a specific Deployment.
• external.metrics.k8s.io — metrics not tied to any Kubernetes object at all, like an SQS queue depth from AWS.

Prometheus Adapter implements the custom.metrics.k8s.io and external.metrics.k8s.io APIs, translating PromQL queries against your existing Prometheus into something the HPA controller can consume through the standard Kubernetes metrics API pattern.

Architecture Flow

HPA controller → queries custom.metrics.k8s.io (or external.metrics.k8s.io) via the Kubernetes API aggregation layer → the aggregation layer routes that request to the Prometheus Adapter pod → Prometheus Adapter translates the request into a PromQL query → queries Prometheus → returns the result back up the chain → HPA uses it to make a scaling decision.

Prerequisites

• A working Prometheus instance already scraping your application metrics
• Helm installed
• Application exposing relevant metrics at a /metrics endpoint (Prometheus format)

Step 1: Instrument Your Application

Before Prometheus Adapter can expose anything useful, your app needs to emit the metric. Example using a Python app with prometheus_client:

from prometheus_client import Gauge, start_http_server

queue_depth = Gauge('queue_pending_messages', 'Number of pending messages in queue')

def update_metric(count):
    queue_depth.set(count)

start_http_server(8000)

Make sure Prometheus is actually scraping it — either via a ServiceMonitor (Prometheus Operator) or static scrape config:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: queue-consumer-metrics
  namespace: production
spec:
  selector:
    matchLabels:
      app: queue-consumer
  endpoints:
  - port: metrics
    interval: 15s

Verify the metric shows up in Prometheus:

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

Query queue_pending_messages in the Prometheus UI to confirm it’s present before moving on.

Step 2: Install Prometheus Adapter

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# values-adapter.yaml
prometheus:
  url: http://prometheus-operated.monitoring.svc
  port: 9090

rules:
  default: false
  custom:
  - seriesQuery: 'queue_pending_messages{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace:
          resource: "namespace"
        pod:
          resource: "pod"
    name:
      matches: "queue_pending_messages"
      as: "queue_pending_messages"
    metricsQuery: 'avg_over_time(<<.Series>>{<<.LabelMatchers>>}[2m])'

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  -n monitoring \
  -f values-adapter.yaml

Setting default: false is deliberate — the chart’s built-in default rules generate a large, often noisy set of auto-discovered metrics. Defining explicit custom rules gives you precise control over exactly what’s exposed to the HPA.

Step 3: Verify the Custom Metrics API

kubectl get apiservices | grep custom.metrics

v1beta1.custom.metrics.k8s.io   monitoring/prometheus-adapter   True

Query the API directly to confirm the metric is actually reachable:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/queue_pending_messages" | jq .

{
  "items": [
    {
      "describedObject": {
        "kind": "Pod",
        "name": "queue-consumer-6f9b8d-xk2p1",
        "namespace": "production"
      },
      "metricName": "queue_pending_messages",
      "value": "47"
    }
  ]
}

If this returns data, the HPA can now consume it.

Step 4: Create an HPA Using the Custom Metric

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: queue-consumer-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-consumer
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: queue_pending_messages
      target:
        type: AverageValue
        averageValue: "10"

This tells the HPA: keep the average queue_pending_messages per pod around 10 — if the queue backs up to 470 messages across pods, the HPA scales out toward roughly 47 replicas (capped at maxReplicas: 20).

kubectl apply -f queue-consumer-hpa.yaml
kubectl get hpa queue-consumer-hpa -n production

NAME                  REFERENCE                    TARGETS   MINPODS   MAXPODS   REPLICAS
queue-consumer-hpa    Deployment/queue-consumer     47/10     2         20        20

External Metrics: Scaling on Non-Kubernetes Signals

For metrics not tied to a specific pod — like total SQS queue depth, or a business metric aggregated across an entire system — use external.metrics.k8s.io instead:

rules:
  external:
  - seriesQuery: 'sqs_queue_depth{queue_name!=""}'
    resources:
      overrides:
        namespace:
          resource: "namespace"
    name:
      matches: "sqs_queue_depth"
      as: "sqs_queue_depth"
    metricsQuery: 'avg_over_time(<<.Series>>{<<.LabelMatchers>>}[2m])'

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sqs-consumer-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sqs-consumer
  minReplicas: 1
  maxReplicas: 30
  metrics:
  - type: External
    external:
      metric:
        name: sqs_queue_depth
        selector:
          matchLabels:
            queue_name: "orders-queue"
      target:
        type: AverageValue
        averageValue: "100"

This assumes something (like yet-another-cloudwatch-exporter or a custom exporter) is publishing SQS metrics into Prometheus in the first place — Prometheus Adapter only exposes what’s already in Prometheus, it doesn’t pull from cloud APIs directly.

Combining Multiple Metrics in One HPA

The HPA evaluates all defined metrics and scales based on whichever demands the most replicas:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "500"

If CPU wants 5 replicas but request rate wants 12, the HPA scales to 12 — it always takes the maximum across all metrics to avoid under-provisioning.

Behavior Tuning: Avoiding Flapping

Custom metrics tend to be noisier than CPU. Use behavior to smooth scaling decisions:

spec:
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 25
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Percent
        value: 100
        periodSeconds: 30

This scales up aggressively (double capacity in 30s if needed) but scales down conservatively (max 25% reduction per minute, with a 5-minute stabilization window) — appropriate when a queue-depth spike needs a fast response but you don’t want to thrash replicas as the queue drains.

Troubleshooting

# Confirm adapter is running and healthy
kubectl get pods -n monitoring -l app=prometheus-adapter
kubectl logs -n monitoring -l app=prometheus-adapter

# Check what metrics the adapter currently exposes
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .

# Check HPA events for scaling decisions or errors
kubectl describe hpa queue-consumer-hpa -n production

Common failure: unable to get metric queue_pending_messages: no metrics returned from custom metrics API — usually means the seriesQuery in the adapter config doesn’t match your actual metric’s labels, or the metric simply isn’t being scraped yet.

Common Mistakes

• Using the default auto-discovery rules in production without review — generates a sprawling, hard-to-audit set of exposed metrics.
• Metric label mismatches between what the adapter’s seriesQuery expects and what Prometheus actually has, causing silent empty results.
• Not tuning behavior, leading to replica flapping on noisy metrics like queue depth that naturally oscillates.
• Forgetting minReplicas appropriate for baseline load — scaling all the way to zero on custom metrics can introduce cold-start latency for the first few requests after a scale-from-zero event (and HPA doesn’t support true scale-to-zero without KEDA).
• Overlooking metric staleness — metricsQuery averaging windows that are too short can make the HPA overreact to brief spikes; too long, and it under-reacts to genuine sustained load.

Debugging seriesQuery Matching Issues

The single most time-consuming part of setting up Prometheus Adapter is usually getting seriesQuery and the label overrides correct on the first try. The adapter’s config supports a discovery mode that helps verify what it’s actually matching before you wire it into an HPA:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[] | select(.name | contains("queue"))'

If this returns nothing, the issue is almost always one of:

1. The seriesQuery label selector doesn’t match your metric’s actual labels — check what Prometheus itself sees with a direct query in the Prometheus UI before assuming the adapter is broken.
2. The metric has no pod or namespace label, so the resources.overrides mapping in your adapter config has nothing to bind to.
3. The adapter pod hasn’t picked up a config change — unlike Prometheus itself, older versions of the adapter don’t hot-reload config changes and need a rollout restart:

kubectl rollout restart deployment/prometheus-adapter -n monitoring

A useful habit is testing new adapter rules against a metric you can control directly (like a Gauge you can set to an arbitrary test value via curl or a debug endpoint) before wiring up the real production metric — it isolates configuration bugs from application-level metric-emission bugs.

Scaling Down to Zero

The standard HPA (even with custom metrics via Prometheus Adapter) cannot scale a Deployment to zero replicas — minReplicas must be at least 1. This is a deliberate Kubernetes API constraint, not a Prometheus Adapter limitation. For workloads that should genuinely scale to zero when idle (a common cost-saving pattern for bursty, infrequently-used services), KEDA’s ScaledObject is the correct tool, since it wraps the HPA but adds its own zero-to-one scaling logic outside the HPA’s normal constraints:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: queue-consumer-scaledobject
  namespace: production
spec:
  scaleTargetRef:
    name: queue-consumer
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-operated.monitoring.svc:9090
      metricName: queue_pending_messages
      threshold: "10"
      query: avg(queue_pending_messages)

Notice KEDA’s prometheus trigger type queries Prometheus directly, without needing Prometheus Adapter as an intermediary at all — for teams already invested in KEDA, this can actually replace the adapter entirely for Prometheus-based scaling triggers.

When to Consider KEDA Instead

For event-driven scaling (queue-based workloads, scale-to-zero, a huge library of pre-built scalers for SQS, Kafka, RabbitMQ, etc.), KEDA is often a more ergonomic choice than hand-rolling Prometheus Adapter rules — it wraps the same HPA mechanism but ships scalers for dozens of external systems out of the box.

Summary

Prometheus Adapter bridges your existing Prometheus metrics into Kubernetes’ custom and external metrics APIs, letting the HPA scale on the signals that actually matter for your workload — queue depth, request rate, business metrics — not just CPU and memory. Define explicit adapter rules rather than relying on auto-discovery, tune HPA behavior to avoid flapping on noisier metrics, and reach for KEDA when your scaling triggers are primarily external event sources.

References

• Kubernetes Documentation: Horizontal Pod Autoscaling
• Prometheus Adapter GitHub Repository
• Kubernetes Custom Metrics API Design
• KEDA Documentation
• CNCF Kubernetes Project