How to Use ResourceQuotas in Kubernetes

How to Use ResourceQuotas in Kubernetes

A shared Kubernetes cluster works well right up until one team’s workload quietly consumes all the CPU, memory, or even object count headroom, leaving everyone else fighting for scraps or, worse, unable to create new resources at all. ResourceQuotas are Kubernetes’ built-in answer to this problem — a namespace-scoped mechanism for capping how much a team or application can consume. In this guide, I’ll go through how ResourceQuotas work, how to configure them properly, and how they interact with other scheduling and limiting mechanisms.

What Problem ResourceQuotas Solve

In a multi-tenant cluster — multiple teams, multiple applications, one shared pool of nodes — there’s nothing stopping a single namespace from requesting unbounded CPU and memory, or creating thousands of ConfigMaps, unless something enforces a limit. A ResourceQuota is an admission-time constraint applied per namespace that caps:

  • Aggregate compute resources (CPU/memory requests and limits) across all Pods in the namespace.
  • Storage (total PVC capacity, count of PVCs).
  • Object counts (how many Pods, Services, Secrets, ConfigMaps, etc. can exist).
  • Extended resources like GPUs.

Kubernetes Architecture: Where Quotas Are Enforced

ResourceQuotas are enforced by an admission controller (ResourceQuota), which runs as part of the API server’s request pipeline, after authentication and authorization (RBAC) but before the object is persisted to etcd:

  1. A request comes in to create a Pod (or any quota-tracked object).
  2. The ResourceQuota admission controller checks the namespace’s current usage against any ResourceQuota objects defined there.
  3. If the request would exceed a quota, it’s rejected outright with a 403 Forbidden — the object is never created.

This is fundamentally different from a PriorityClass or Limit Range — quota enforcement happens at admission time, before scheduling is even considered.

Step 1: Define a Basic Compute ResourceQuota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
kubectl apply -f team-a-quota.yaml
kubectl describe resourcequota team-a-quota -n team-a

Output:

Name:            team-a-quota
Namespace:       team-a
Resource         Used  Hard
--------         ----  ----
limits.cpu       0     20
limits.memory    0     40Gi
requests.cpu     0     10
requests.memory  0     20Gi

Now, any attempt to create a Pod without explicit resource requests/limits will actually fail in this namespace — once a ResourceQuota tracks compute resources, every Pod must declare them explicitly. This is a very common gotcha worth knowing up front.

Step 2: Combine with a LimitRange for Sane Defaults

Since ResourceQuota requires explicit requests/limits on every Pod once compute quota exists, pair it with a LimitRange so teams aren’t forced to manually specify resources on every single container:

apiVersion: v1
kind: LimitRange
metadata:
  name: team-a-limits
  namespace: team-a
spec:
  limits:
    - type: Container
      defaultRequest:
        cpu: 250m
        memory: 256Mi
      default:
        cpu: 500m
        memory: 512Mi
      max:
        cpu: "2"
        memory: 4Gi
      min:
        cpu: 50m
        memory: 64Mi
kubectl apply -f team-a-limits.yaml

Now any Pod created without explicit resources automatically gets the defaults injected, and the LimitRange also enforces sane per-container bounds independent of the namespace-wide quota.

Step 3: Object Count Quotas

Cap how many objects a namespace can create — useful for preventing runaway automation or misconfigured CI pipelines from flooding the cluster:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-object-counts
  namespace: team-a
spec:
  hard:
    pods: "50"
    services: "10"
    services.loadbalancers: "2"
    persistentvolumeclaims: "20"
    secrets: "30"
    configmaps: "30"

services.loadbalancers is worth calling out specifically — LoadBalancer Services often cost real money via cloud provider infrastructure, so capping them prevents accidental cost overruns from a namespace spinning up dozens of external load balancers.

Step 4: Storage Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-storage-quota
  namespace: team-a
spec:
  hard:
    requests.storage: 500Gi
    persistentvolumeclaims: "20"
    fast-ssd.storageclass.storage.k8s.io/requests.storage: 100Gi

That last line demonstrates StorageClass-scoped quotas — you can cap usage per storage class, so a team can use as much cheap standard storage as needed but is capped tightly on expensive fast-SSD storage.

Step 5: Scoped Quotas by Priority Class

You can scope a ResourceQuota to only apply to Pods of a certain priority, which is powerful when combined with Priority and Preemption — for example, capping how much high-priority capacity any one namespace can consume, so no single team can monopolize preemption power:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-high-priority-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
  scopeSelector:
    matchExpressions:
      - operator: In
        scopeName: PriorityClass
        values: ["high-priority"]

Step 6: Checking Usage

kubectl get resourcequota -n team-a
kubectl describe resourcequota -n team-a

For a quick view across all namespaces (useful for a platform team doing capacity planning):

kubectl get resourcequota --all-namespaces -o custom-columns=\
NAMESPACE:.metadata.namespace,NAME:.metadata.name,USED_CPU:.status.used.requests\\.cpu,HARD_CPU:.status.hard.requests\\.cpu

What Happens When a Quota Is Exceeded

The API server rejects the request immediately with a clear error:

Error from server (Forbidden): error when creating "pod.yaml": pods "worker-5" is forbidden: exceeded quota: team-a-quota, requested: requests.cpu=2, used: requests.cpu=9, limited: requests.cpu=10

This tells you exactly which quota, which resource, and by how much — extremely useful for debugging CI pipeline failures where a deploy suddenly starts failing after previously working fine (usually because cumulative usage crossed the threshold).

Best Practices

  • Always pair ResourceQuota with LimitRange — without defaults, quota enforcement becomes a constant source of confusing failures for developers who never specified resource requests.
  • Set quotas per-namespace as part of namespace provisioning automation, not as an afterthought — bake it into your namespace-creation pipeline (Terraform, Crossplane, or a platform Operator) so no namespace exists without one.
  • Monitor quota utilization proactively (via Prometheus’s kube-state-metrics, which exposes kube_resourcequota metrics) and alert before teams hit the ceiling, not after they’re blocked mid-deploy.
  • Use StorageClass-scoped and PriorityClass-scoped quotas for fine-grained control rather than one blunt namespace-wide number.
  • Review and adjust quotas periodically as teams’ actual usage patterns become clear — an initial guess is rarely the right long-term number.

Common Mistakes

  • Setting a compute ResourceQuota without a LimitRange, causing every Pod creation to fail with cryptic “must specify limits.cpu” errors until developers figure out why.
  • Forgetting that limits.cpu/limits.memory and requests.cpu/requests.memory are tracked separately — a quota on requests alone doesn’t cap how high limits can go, which matters for burstable workloads.
  • Setting object count quotas too low for legitimate CI/CD churn (e.g., a pods: "10" quota that blocks a rolling deployment that briefly needs old and new replicas simultaneously).
  • Not accounting for quota when debugging “why won’t my Deployment scale up” — quota rejections don’t show up as scheduling failures; they show up as failed object creation, which look different in events.

Disaster Recovery and Multi-Tenancy Considerations

ResourceQuotas are part of your cluster’s declarative state and should be version-controlled alongside your namespace manifests (GitOps), so recreating a cluster or namespace from scratch restores the same guardrails automatically — never rely on manually re-applying quotas after an incident, since a namespace briefly without quota enforcement is a real risk in a busy multi-tenant cluster.

Summary

ResourceQuotas are how Kubernetes keeps a shared, multi-tenant cluster fair — capping compute, storage, and object counts per namespace, enforced right at admission time before anything even reaches the scheduler. Pair them with LimitRanges for sane defaults, use scoped quotas for fine-grained control over priority classes and storage classes, and bake quota provisioning into your namespace automation so every team operates within limits from day one, not as an emergency fix after someone’s already eaten the whole cluster.

References

Total
6
Shares

Leave a Reply

Previous Post
How to Set Up Pod Disruption Budgets with Prometheus in Kubernetes

How to Set Up Pod Disruption Budgets with Prometheus in Kubernetes

Next Post
How to Set Up Kubernetes Monitoring with Zabbix

How to Set Up Kubernetes Monitoring with Zabbix

Related Posts