There’s an important distinction that trips up a lot of people building their first monitoring stack: metrics about resource usage (CPU, memory) are not the same as metrics about object state (is this Deployment fully rolled out, how many replicas are unavailable, when did this Pod last restart). kube-state-metrics exists specifically for the second category, and it’s the piece that turns Prometheus/Grafana from “shows CPU graphs” into “actually understands what Kubernetes objects are doing.”
kube-state-metrics vs metrics-server: A Critical Distinction
These are commonly confused, but they serve entirely different purposes:
- metrics-server: real-time CPU/memory resource usage for Pods and Nodes, used by the Horizontal Pod Autoscaler. It does not persist history.
- kube-state-metrics: structural state of Kubernetes objects — Deployment replica counts, Pod status, Job completion state, PVC status — exposed as Prometheus-format metrics, meant to be scraped and stored over time.
Neither one replaces the other; a complete observability stack uses both.
Installing kube-state-metrics
If installed as part of kube-prometheus-stack (as shown in the Grafana monitoring setup), it’s already running. Standalone installation via Helm:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-state-metrics prometheus-community/kube-state-metrics \
--namespace monitoring --create-namespace
Verify:
kubectl -n monitoring get pods -l app.kubernetes.io/name=kube-state-metrics
NAME READY STATUS RESTARTS
kube-state-metrics-6d7f9c8b7f-abcde 1/1 Running 0
What It Actually Exposes
kube-state-metrics scrapes the Kubernetes API and exposes object state as plain-text Prometheus metrics:
kubectl -n monitoring port-forward svc/kube-state-metrics 8080:8080
curl localhost:8080/metrics | grep kube_deployment
Sample output:
kube_deployment_status_replicas{deployment="myapp",namespace="production"} 3
kube_deployment_status_replicas_available{deployment="myapp",namespace="production"} 2
kube_deployment_status_replicas_unavailable{deployment="myapp",namespace="production"} 1
kube_deployment_spec_replicas{deployment="myapp",namespace="production"} 3
That single set of metrics already tells you something metrics-server never could: this Deployment wants 3 replicas, has 3 total, but only 2 are actually available — a rollout or health-check problem, distinct from a resource-usage problem.
RBAC Required
kube-state-metrics needs broad read-only access across the cluster to see object state everywhere:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: kube-state-metrics
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets", "daemonsets"]
verbs: ["list", "watch"]
- apiGroups: [""]
resources: ["pods", "nodes", "persistentvolumeclaims", "services"]
verbs: ["list", "watch"]
- apiGroups: ["batch"]
resources: ["jobs", "cronjobs"]
verbs: ["list", "watch"]
Note: list/watch only — kube-state-metrics never writes anything, which is worth confirming during any security review.
Connecting to Prometheus
A ServiceMonitor (if using the Prometheus Operator) wires it in automatically:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
selector:
matchLabels:
app.kubernetes.io/name: kube-state-metrics
endpoints:
- port: http-metrics
interval: 30s
Useful PromQL Queries
# Deployments with fewer available replicas than desired
kube_deployment_status_replicas_available
< kube_deployment_spec_replicas
# Pods stuck in a non-Running phase for longer than 5 minutes
kube_pod_status_phase{phase!="Running"} == 1
# PVCs not yet bound
kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
# Nodes reporting NotReady
kube_node_status_condition{condition="Ready", status="true"} == 0
# Jobs that failed
kube_job_status_failed > 0
Alerting on State Metrics
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: deployment-health
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
groups:
- name: deployment-state
rules:
- alert: DeploymentReplicasMismatch
expr: |
kube_deployment_status_replicas_available
< kube_deployment_spec_replicas
for: 10m
labels:
severity: warning
annotations:
summary: "Deployment {{ $labels.deployment }} has fewer replicas than desired"
description: "Namespace {{ $labels.namespace }} — check rollout status"
- alert: PersistentVolumeClaimPending
expr: kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "PVC {{ $labels.persistentvolumeclaim }} stuck Pending"
kubectl apply -f deployment-health-rule.yaml
Grafana Dashboards Built on State Metrics
kube-state-metrics is the primary data source behind most “Kubernetes Cluster Overview” community dashboards (like dashboard ID 315, referenced in the Grafana setup). A minimal custom panel query for a Deployment health table:
sum by (deployment, namespace) (kube_deployment_status_replicas_available)
Label and Annotation Metrics
kube-state-metrics can also expose Kubernetes labels/annotations as metric labels, useful for slicing dashboards by team or cost center:
helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
--namespace monitoring \
--set metricLabelsAllowlist="pods=[team],deployments=[team]"
This surfaces a team label on Pod/Deployment metrics directly from Kubernetes object labels, letting you build per-team dashboards without extra tagging infrastructure.
Resource Considerations
kube-state-metrics itself is lightweight but its memory footprint scales with cluster object count — very large clusters (thousands of Pods/Deployments) should set explicit resource requests/limits and monitor it like any other workload:
resources:
requests:
cpu: 100m
memory: 200Mi
limits:
cpu: 250m
memory: 512Mi
Sharding for Very Large Clusters
On clusters with tens of thousands of objects, a single kube-state-metrics instance listing and watching everything cluster-wide can become a bottleneck and a memory concern. Sharding splits the workload across multiple replicas, each responsible for a subset of objects (typically partitioned by a hash of namespace/name):
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
replicas: 4
serviceName: kube-state-metrics
template:
spec:
containers:
- name: kube-state-metrics
image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0
args:
- --shard=$(SHARD_INDEX)
- --total-shards=4
env:
- name: SHARD_INDEX
valueFrom:
fieldRef:
fieldPath: metadata.labels['apps.kubernetes.io/pod-index']
Prometheus then scrapes all shard Pods and the results naturally combine into a complete view when aggregated in PromQL queries, since each shard exposes a disjoint subset of objects rather than overlapping or duplicate data.
Metrics for Custom Resources
Beyond core Kubernetes objects, kube-state-metrics can be configured to expose metrics for CustomResourceDefinitions via its Custom Resource State feature — useful when an operator-managed resource (like an Argo CD Application or a cert-manager Certificate) has status fields worth turning into dashboards and alerts without writing a dedicated exporter:
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-state-metrics-crs-config
namespace: monitoring
data:
config.yaml: |
spec:
resources:
- groupVersionKind:
group: cert-manager.io
version: v1
kind: Certificate
metrics:
- name: cert_manager_certificate_ready
help: "Certificate ready status"
each:
type: Gauge
gauge:
path: [status, conditions]
labelsFromPath:
type: [type]
valueFrom: [status]
helm upgrade kube-state-metrics prometheus-community/kube-state-metrics \
--namespace monitoring \
--set-file customResourceState.config=config.yaml
This turns something like “is this cert-manager Certificate actually ready” — previously visible only via kubectl describe — into a first-class Prometheus metric that can be dashboarded and alerted on exactly like any built-in Kubernetes object state.
Why Not Just Use kubectl for This?
A reasonable question: kubectl get deployment already shows replica counts, so why does a separate metrics exporter matter? The answer is history and alerting. kubectl gives a snapshot of right now — useful for active debugging, useless for answering “how many times did this Deployment drop below its desired replica count last week” or for triggering an alert the moment it happens rather than whenever someone happens to run the command. kube-state-metrics turns that same information into a continuously scraped time series, which is what makes both historical analysis and automated alerting possible in the first place — the underlying data is the same; what changes is whether it’s captured over time or only visible in the instant someone looks.
Where kube-state-metrics Fits in a Full Observability Pipeline
It’s worth situating this specific tool within the broader observability stack, since a common early mistake is treating any single component as sufficient on its own:
- kube-state-metrics — structural object state (this article’s focus).
- metrics-server — real-time resource usage, feeding the Horizontal Pod Autoscaler.
- cAdvisor (built into the kubelet) — per-container resource usage at a finer granularity than metrics-server exposes, often scraped directly by Prometheus.
- Application metrics — whatever the application itself exposes (request latency, business-logic counters), unrelated to Kubernetes object state entirely.
A genuinely complete monitoring setup scrapes all four categories into the same Prometheus instance, letting a single Grafana dashboard correlate “the Deployment lost a replica” (kube-state-metrics) with “that Pod was OOMKilled” (cAdvisor/kubelet) and “error rate on that endpoint spiked at the same moment” (application metrics) — three separate data sources telling one coherent story about a single incident.
Cardinality: The Hidden Cost of Exposing Every Label
It’s tempting, once metricLabelsAllowlist is discovered, to allowlist every label an organization uses across every resource type — but this has a real, sometimes severe cost. Each unique combination of label values on a metric creates a distinct time series in Prometheus, and high-cardinality labels (anything with many unique values, like a commit-sha or pod-name label applied broadly) can multiply the number of stored time series by orders of magnitude, degrading Prometheus query performance and inflating storage requirements substantially.
# Check current time series count contributed by kube-state-metrics
curl -s localhost:9090/api/v1/query \
--data-urlencode 'query=count({__name__=~"kube_.+"})'
A reasonable practice: allowlist only labels genuinely needed for dashboarding or alerting (team, environment, cost-center) and deliberately avoid anything with unbounded or near-unbounded cardinality. If a specific high-cardinality label is genuinely needed for a narrow use case, scoping the allowlist to only the specific resource type that needs it — rather than applying it globally across every metric kube-state-metrics exposes — keeps the blast radius of that decision contained.
Retention and Downsampling Considerations
Because kube-state-metrics metrics are typically scraped at a fairly short interval (15-30 seconds is common) to catch state transitions promptly, the resulting data volume adds up over longer retention windows. For clusters using Thanos or a similar long-term storage layer, configuring downsampling rules specifically for kube-state-metrics-derived series — reducing to 5-minute resolution after a day, for instance — keeps months of history queryable without the storage cost of full-resolution data indefinitely, while still preserving enough fidelity to answer “how has replica availability trended over the last quarter” style questions that don’t need second-level precision to be useful.
Common Mistakes
- Confusing kube-state-metrics with metrics-server and wondering why CPU/memory graphs aren’t showing up from it — they never will; that’s not its job.
- Alerting on raw Pod restart counts instead of
increase()over a time window, producing meaningless static thresholds. - Not scoping
metricLabelsAllowlist, which can produce extremely high cardinality metrics on clusters with many unique label values. - Forgetting that kube-state-metrics reflects the API server’s view of desired/actual state — it won’t catch application-level bugs where a Pod reports
Runningbut the app inside is actually broken; that requires application metrics, not state metrics.
Summary
kube-state-metrics fills the gap between “the cluster is technically running” and “the cluster is running the way you intended.” It answers structural questions — replica counts, rollout progress, PVC binding status, Job completion — that resource-usage metrics alone can’t. Paired with Prometheus alerting rules, it’s often the earliest signal that a deployment or a piece of infrastructure state has quietly drifted from what was intended.
References
- kube-state-metrics documentation: https://github.com/kubernetes/kube-state-metrics
- Metrics reference: https://github.com/kubernetes/kube-state-metrics/tree/main/docs
- Prometheus Operator: https://prometheus-operator.dev/docs/getting-started/introduction/
- Kubernetes API reference: https://kubernetes.io/docs/reference/kubernetes-api/