How to Set Up Custom Metrics in Kubernetes

CPU and memory utilization are the default signals the Horizontal Pod Autoscaler understands out of the box, but they’re often the wrong signal entirely. If you’re running a queue worker, what actually matters is queue depth. If you’re running an API, it might be requests-per-second or p99 latency. Scaling on CPU alone in these cases means you’re reacting to a proxy metric instead of the thing you actually care about. This guide walks through setting up custom and external metrics for Kubernetes autoscaling, end-to-end, on AWS EKS.

The Kubernetes Metrics APIs — Three Different Things

This is the part that confuses people first, so let’s get it straight before writing any YAML. Kubernetes’ autoscaling ecosystem is built on three distinct metrics APIs:

1. metrics.k8s.io (Resource Metrics API) — served by metrics-server, provides basic CPU/memory usage. This is what powers default HPA behavior.
2. custom.metrics.k8s.io — metrics associated with Kubernetes objects (e.g., “requests per second for this specific Service,” “queue depth label matching this Deployment”). Requires a custom metrics adapter.
3. external.metrics.k8s.io — metrics that come from outside the cluster entirely and aren’t tied to any Kubernetes object (e.g., an SQS queue depth, a CloudWatch metric, a metric from a third-party SaaS).

The HPA can consume metrics from any of these three APIs simultaneously, in a single HorizontalPodAutoscaler object.

Installing metrics-server (Baseline)

Even if your goal is custom metrics, metrics-server is still worth having for baseline CPU/memory visibility and kubectl top:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl get deployment metrics-server -n kube-system
kubectl top nodes
kubectl top pods -A

Setting Up the Prometheus Adapter for Custom Metrics

If you’re already running Prometheus (see the companion Prometheus monitoring article), the most common path to custom.metrics.k8s.io is the Prometheus Adapter, which translates PromQL queries into the custom metrics API that the HPA controller understands.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# prometheus-adapter-values.yaml
prometheus:
  url: http://kube-prometheus-stack-prometheus.monitoring.svc
  port: 9090

rules:
  default: false
  custom:
    - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: { resource: "namespace" }
          pod: { resource: "pod" }
      name:
        matches: "http_requests_total"
        as: "http_requests_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  -n monitoring -f prometheus-adapter-values.yaml

Verify the custom metrics API is being served:

kubectl get apiservices | grep custom.metrics

v1beta1.custom.metrics.k8s.io   monitoring/prometheus-adapter   True   5m

Query it directly to sanity check:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/production/pods/*/http_requests_per_second" | jq .

HPA Scaling on a Custom Metric

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-custom
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"

This scales web-app to keep average requests-per-second-per-pod around 100 — a far more direct signal of actual load than CPU, especially for I/O-bound services where CPU usage doesn’t correlate well with request volume.

kubectl apply -f web-app-custom-hpa.yaml
kubectl get hpa web-app-custom -n production

NAME              REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS
web-app-custom    Deployment/web-app      85/100 (avg)    3         50        4

Combining Multiple Metrics in One HPA

The HPA evaluates all listed metrics and scales to satisfy the largest recommendation among them:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-multi-metric
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"
    - type: Pods
      pods:
        metric:
          name: http_request_duration_p99_seconds
        target:
          type: AverageValue
          averageValue: "500m"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

The behavior block is worth calling out — asymmetric scale-up/scale-down policies are a real production best practice: scale up aggressively (don’t leave users waiting during a traffic spike) but scale down conservatively (avoid flapping, since terminating pods too eagerly during a brief lull just means re-provisioning them minutes later).

External Metrics: Scaling on SQS Queue Depth (AWS-Specific)

This is one of the most common real-world custom autoscaling needs on EKS: scaling worker pods based on the depth of an SQS queue they’re consuming from — a metric that has no natural association with any Kubernetes object, since it lives entirely in AWS.

The cleanest modern approach uses KEDA (Kubernetes Event-Driven Autoscaling), a CNCF project purpose-built for exactly this class of problem, supporting dozens of external event sources out of the box.

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-worker-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: sqs-worker
  minReplicaCount: 0
  maxReplicaCount: 30
  cooldownPeriod: 120
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-aws-credentials
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789012/worker-queue
        queueLength: "5"
        awsRegion: us-east-1

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-aws-credentials
  namespace: production
spec:
  podIdentity:
    provider: aws-eks

apiVersion: v1
kind: ServiceAccount
metadata:
  name: keda-operator
  namespace: keda
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/keda-sqs-reader

With minReplicaCount: 0, KEDA gives you true scale-to-zero — when the queue is empty, there are no worker pods running at all, and KEDA spins them up the moment messages appear. This is something the standard HPA fundamentally cannot do (it requires at least 1 replica), and it’s a genuinely significant cost optimization for bursty batch/queue workloads.

kubectl apply -f sqs-scaledobject.yaml
kubectl get scaledobject sqs-worker-scaler -n production

NAME                READY   ACTIVE   MIN   MAX   TRIGGERS
sqs-worker-scaler   True    True     0     30    aws-sqs-queue

Under the hood, KEDA creates and manages a standard HPA object for you, backed by its own metrics adapter serving the external.metrics.k8s.io API:

kubectl get hpa -n production

Scaling on CloudWatch Metrics via KEDA

For metrics that live purely in CloudWatch — like an RDS connection count, an ALB request count, or a custom application metric you’re already publishing to CloudWatch:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: cloudwatch-scaler
  namespace: production
spec:
  scaleTargetRef:
    name: web-app
  minReplicaCount: 2
  maxReplicaCount: 20
  triggers:
    - type: aws-cloudwatch
      authenticationRef:
        name: keda-aws-credentials
      metadata:
        namespace: AWS/ApplicationELB
        metricName: RequestCountPerTarget
        dimensionName: TargetGroup
        dimensionValue: targetgroup/web-app-tg/xxxxxxxxxxxxxxxx
        targetMetricValue: "50"
        minMetricValue: "0"
        awsRegion: us-east-1

Custom Metrics with the Vertical Pod Autoscaler

Worth a brief mention: the HPA scales replica count, but sometimes what you actually need is to right-size a single pod’s resource requests over time — that’s the Vertical Pod Autoscaler’s job, and it’s complementary, not competing, with HPA (though running both on CPU/memory for the same workload simultaneously is explicitly discouraged, since they can fight each other):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-app-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  updatePolicy:
    updateMode: "Off"  # recommendation-only mode, doesn't auto-apply

Starting in "Off" mode and just observing VPA’s recommendations before ever enabling "Auto" is the safer rollout pattern — auto-applied resource changes can trigger unexpected pod restarts.

Monitoring the Autoscaling Pipeline Itself

# Watch HPA decisions and events in real time
kubectl describe hpa web-app-custom -n production

# Check the custom metrics adapter is healthy
kubectl logs -n monitoring -l app=prometheus-adapter --tail=50

# Check KEDA operator logs for scaler issues
kubectl logs -n keda -l app=keda-operator --tail=50

Useful Prometheus queries for observing autoscaler behavior over time:

kube_horizontalpodautoscaler_status_current_replicas
kube_horizontalpodautoscaler_status_desired_replicas

Troubleshooting

HPA shows <unknown> for a custom metric target:

kubectl describe hpa web-app-custom -n production

Almost always means the custom metrics adapter isn’t serving that metric — check the adapter’s PromQL query actually returns data (http_requests_total might not exist yet if no traffic has hit the service), or the metric name doesn’t match what’s registered.

KEDA ScaledObject stuck with ACTIVE: False:

kubectl describe scaledobject sqs-worker-scaler -n production

Usually an IAM permissions issue (the keda-operator ServiceAccount’s IAM role lacking sqs:GetQueueAttributes), or the queue URL/region being wrong.

Common Mistakes

• Scaling purely on CPU for I/O-bound or queue-driven workloads, chasing a proxy metric instead of the real signal.
• Running HPA and VPA on the same resource (CPU/memory) for the same workload simultaneously, causing scaling conflicts.
• Forgetting minReplicaCount: 0 requires KEDA (standard HPA can’t scale to zero), and being surprised idle workers still cost money.
• Overly aggressive scale-down stabilizationWindowSeconds, causing replica count flapping under bursty-but-brief traffic patterns.
• Not testing custom metric queries directly against Prometheus before wiring them into an HPA — debugging “why isn’t this HPA scaling” is much harder than debugging a raw PromQL query.

Best Practices

• Use KEDA for anything event-driven or external (queues, CloudWatch metrics, Kafka lag) rather than hand-rolling a custom metrics adapter for every source.
• Use asymmetric HPA behavior policies — aggressive scale-up, conservative scale-down.
• Validate custom metric queries independently in Prometheus/CloudWatch before wiring them into autoscaling objects.
• Combine multiple signals (CPU + custom request rate + latency) in one HPA when a single metric doesn’t fully capture load.
• Start VPA in recommendation-only mode before ever enabling automatic updates.

Summary

Kubernetes’ autoscaling story extends well beyond CPU and memory once you bring in the custom.metrics.k8s.io and external.metrics.k8s.io APIs. The Prometheus Adapter unlocks scaling on any metric you’re already collecting in Prometheus — request rates, latency, business metrics — while KEDA extends this to external, non-Kubernetes-native sources like SQS queue depth or CloudWatch metrics, with genuine scale-to-zero support standard HPA can’t offer. On AWS EKS, pairing KEDA with IRSA or Pod Identity for AWS API access gives you a clean, production-grade path to scaling workloads on the metrics that actually matter for your application, not just the ones Kubernetes happens to track by default.

References

• Kubernetes Documentation: Horizontal Pod Autoscaling
• Kubernetes Documentation: HPA Walkthrough with Custom Metrics
• Prometheus Adapter GitHub Repository
• KEDA Documentation
• Kubernetes Vertical Pod Autoscaler GitHub Repository
• AWS EKS Best Practices Guide: Autoscaling