How to Use State Metrics in Kubernetes

There’s an important distinction that trips up a lot of people building their first monitoring stack: metrics about resource usage (CPU, memory) are not the same as metrics about object state (is this Deployment fully rolled out, how many replicas are unavailable, when did this Pod last restart). kube-state-metrics exists specifically for the second category, and it’s the piece that turns Prometheus/Grafana from “shows CPU graphs” into “actually understands what Kubernetes objects are doing.”

kube-state-metrics vs metrics-server: A Critical Distinction

These are commonly confused, but they serve entirely different purposes:

• metrics-server: real-time CPU/memory resource usage for Pods and Nodes, used by the Horizontal Pod Autoscaler. It does not persist history.
• kube-state-metrics: structural state of Kubernetes objects — Deployment replica counts, Pod status, Job completion state, PVC status — exposed as Prometheus-format metrics, meant to be scraped and stored over time.

Neither one replaces the other; a complete observability stack uses both.

Installing kube-state-metrics

If installed as part of kube-prometheus-stack (as shown in the Grafana monitoring setup), it’s already running. Standalone installation via Helm:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring --create-namespace

Verify:

kubectl -n monitoring get pods -l app.kubernetes.io/name=kube-state-metrics

NAME                                  READY   STATUS    RESTARTS
kube-state-metrics-6d7f9c8b7f-abcde   1/1     Running   0

What It Actually Exposes

kube-state-metrics scrapes the Kubernetes API and exposes object state as plain-text Prometheus metrics:

kubectl -n monitoring port-forward svc/kube-state-metrics 8080:8080
curl localhost:8080/metrics | grep kube_deployment

Sample output:

kube_deployment_status_replicas{deployment="myapp",namespace="production"} 3
kube_deployment_status_replicas_available{deployment="myapp",namespace="production"} 2
kube_deployment_status_replicas_unavailable{deployment="myapp",namespace="production"} 1
kube_deployment_spec_replicas{deployment="myapp",namespace="production"} 3

That single set of metrics already tells you something metrics-server never could: this Deployment wants 3 replicas, has 3 total, but only 2 are actually available — a rollout or health-check problem, distinct from a resource-usage problem.

RBAC Required

kube-state-metrics needs broad read-only access across the cluster to see object state everywhere:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
    verbs: ["list", "watch"]
  - apiGroups: [""]
    resources: ["pods", "nodes", "persistentvolumeclaims", "services"]
    verbs: ["list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["list", "watch"]

Note: list/watch only — kube-state-metrics never writes anything, which is worth confirming during any security review.

Connecting to Prometheus

A ServiceMonitor (if using the Prometheus Operator) wires it in automatically:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  endpoints:
    - port: http-metrics
      interval: 30s

Useful PromQL Queries

# Deployments with fewer available replicas than desired
kube_deployment_status_replicas_available
  < kube_deployment_spec_replicas

# Pods stuck in a non-Running phase for longer than 5 minutes
kube_pod_status_phase{phase!="Running"} == 1

# PVCs not yet bound
kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1

# Nodes reporting NotReady
kube_node_status_condition{condition="Ready", status="true"} == 0

# Jobs that failed
kube_job_status_failed > 0

Alerting on State Metrics

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: deployment-health
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: deployment-state
      rules:
        - alert: DeploymentReplicasMismatch
          expr: |
            kube_deployment_status_replicas_available
              < kube_deployment_spec_replicas
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Deployment {{ $labels.deployment }} has fewer replicas than desired"
            description: "Namespace {{ $labels.namespace }} — check rollout status"

        - alert: PersistentVolumeClaimPending
          expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "PVC {{ $labels.persistentvolumeclaim }} stuck Pending"

kubectl apply -f deployment-health-rule.yaml

Grafana Dashboards Built on State Metrics

kube-state-metrics is the primary data source behind most “Kubernetes Cluster Overview” community dashboards (like dashboard ID 315, referenced in the Grafana setup). A minimal custom panel query for a Deployment health table:

sum by (deployment, namespace) (kube_deployment_status_replicas_available)

Label and Annotation Metrics

kube-state-metrics can also expose Kubernetes labels/annotations as metric labels, useful for slicing dashboards by team or cost center:

helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --set metricLabelsAllowlist="pods=[team],deployments=[team]"

This surfaces a team label on Pod/Deployment metrics directly from Kubernetes object labels, letting you build per-team dashboards without extra tagging infrastructure.

Resource Considerations

kube-state-metrics itself is lightweight but its memory footprint scales with cluster object count — very large clusters (thousands of Pods/Deployments) should set explicit resource requests/limits and monitor it like any other workload:

resources:
  requests:
    cpu: 100m
    memory: 200Mi
  limits:
    cpu: 250m
    memory: 512Mi

Sharding for Very Large Clusters

On clusters with tens of thousands of objects, a single kube-state-metrics instance listing and watching everything cluster-wide can become a bottleneck and a memory concern. Sharding splits the workload across multiple replicas, each responsible for a subset of objects (typically partitioned by a hash of namespace/name):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 4
  serviceName: kube-state-metrics
  template:
    spec:
      containers:
        - name: kube-state-metrics
          image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0
          args:
            - --shard=$(SHARD_INDEX)
            - --total-shards=4
          env:
            - name: SHARD_INDEX
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['apps.kubernetes.io/pod-index']

Prometheus then scrapes all shard Pods and the results naturally combine into a complete view when aggregated in PromQL queries, since each shard exposes a disjoint subset of objects rather than overlapping or duplicate data.

Metrics for Custom Resources

Beyond core Kubernetes objects, kube-state-metrics can be configured to expose metrics for CustomResourceDefinitions via its Custom Resource State feature — useful when an operator-managed resource (like an Argo CD Application or a cert-manager Certificate) has status fields worth turning into dashboards and alerts without writing a dedicated exporter:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-state-metrics-crs-config
  namespace: monitoring
data:
  config.yaml: |
    spec:
      resources:
        - groupVersionKind:
            group: cert-manager.io
            version: v1
            kind: Certificate
          metrics:
            - name: cert_manager_certificate_ready
              help: "Certificate ready status"
              each:
                type: Gauge
                gauge:
                  path: [status, conditions]
                  labelsFromPath:
                    type: [type]
                  valueFrom: [status]

helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
  --namespace monitoring \
  --set-file customResourceState.config=config.yaml

This turns something like “is this cert-manager Certificate actually ready” — previously visible only via kubectl describe — into a first-class Prometheus metric that can be dashboarded and alerted on exactly like any built-in Kubernetes object state.

Why Not Just Use kubectl for This?

A reasonable question: kubectl get deployment already shows replica counts, so why does a separate metrics exporter matter? The answer is history and alerting. kubectl gives a snapshot of right now — useful for active debugging, useless for answering “how many times did this Deployment drop below its desired replica count last week” or for triggering an alert the moment it happens rather than whenever someone happens to run the command. kube-state-metrics turns that same information into a continuously scraped time series, which is what makes both historical analysis and automated alerting possible in the first place — the underlying data is the same; what changes is whether it’s captured over time or only visible in the instant someone looks.

Where kube-state-metrics Fits in a Full Observability Pipeline

It’s worth situating this specific tool within the broader observability stack, since a common early mistake is treating any single component as sufficient on its own:

• kube-state-metrics — structural object state (this article’s focus).
• metrics-server — real-time resource usage, feeding the Horizontal Pod Autoscaler.
• cAdvisor (built into the kubelet) — per-container resource usage at a finer granularity than metrics-server exposes, often scraped directly by Prometheus.
• Application metrics — whatever the application itself exposes (request latency, business-logic counters), unrelated to Kubernetes object state entirely.

A genuinely complete monitoring setup scrapes all four categories into the same Prometheus instance, letting a single Grafana dashboard correlate “the Deployment lost a replica” (kube-state-metrics) with “that Pod was OOMKilled” (cAdvisor/kubelet) and “error rate on that endpoint spiked at the same moment” (application metrics) — three separate data sources telling one coherent story about a single incident.

Cardinality: The Hidden Cost of Exposing Every Label

It’s tempting, once metricLabelsAllowlist is discovered, to allowlist every label an organization uses across every resource type — but this has a real, sometimes severe cost. Each unique combination of label values on a metric creates a distinct time series in Prometheus, and high-cardinality labels (anything with many unique values, like a commit-sha or pod-name label applied broadly) can multiply the number of stored time series by orders of magnitude, degrading Prometheus query performance and inflating storage requirements substantially.

# Check current time series count contributed by kube-state-metrics
curl -s localhost:9090/api/v1/query \
  --data-urlencode 'query=count({__name__=~"kube_.+"})'

A reasonable practice: allowlist only labels genuinely needed for dashboarding or alerting (team, environment, cost-center) and deliberately avoid anything with unbounded or near-unbounded cardinality. If a specific high-cardinality label is genuinely needed for a narrow use case, scoping the allowlist to only the specific resource type that needs it — rather than applying it globally across every metric kube-state-metrics exposes — keeps the blast radius of that decision contained.

Retention and Downsampling Considerations

Because kube-state-metrics metrics are typically scraped at a fairly short interval (15-30 seconds is common) to catch state transitions promptly, the resulting data volume adds up over longer retention windows. For clusters using Thanos or a similar long-term storage layer, configuring downsampling rules specifically for kube-state-metrics-derived series — reducing to 5-minute resolution after a day, for instance — keeps months of history queryable without the storage cost of full-resolution data indefinitely, while still preserving enough fidelity to answer “how has replica availability trended over the last quarter” style questions that don’t need second-level precision to be useful.

Common Mistakes

• Confusing kube-state-metrics with metrics-server and wondering why CPU/memory graphs aren’t showing up from it — they never will; that’s not its job.
• Alerting on raw Pod restart counts instead of increase() over a time window, producing meaningless static thresholds.
• Not scoping metricLabelsAllowlist, which can produce extremely high cardinality metrics on clusters with many unique label values.
• Forgetting that kube-state-metrics reflects the API server’s view of desired/actual state — it won’t catch application-level bugs where a Pod reports Running but the app inside is actually broken; that requires application metrics, not state metrics.

Summary

kube-state-metrics fills the gap between “the cluster is technically running” and “the cluster is running the way you intended.” It answers structural questions — replica counts, rollout progress, PVC binding status, Job completion — that resource-usage metrics alone can’t. Paired with Prometheus alerting rules, it’s often the earliest signal that a deployment or a piece of infrastructure state has quietly drifted from what was intended.

References

• kube-state-metrics documentation: https://github.com/kubernetes/kube-state-metrics
• Metrics reference: https://github.com/kubernetes/kube-state-metrics/tree/main/docs
• Prometheus Operator: https://prometheus-operator.dev/docs/getting-started/introduction/
• Kubernetes API reference: https://kubernetes.io/docs/reference/kubernetes-api/