How to Implement Resource Quotas in Kubernetes

How to Implement Resource Quotas in Kubernetes

I once watched a single misconfigured Deployment — a bad replicas value combined with no resource limits — quietly consume enough CPU and memory across a shared cluster that three unrelated teams started seeing their pods get evicted. Nobody had done anything wrong except forget a safeguard that should have been there from day one: a ResourceQuota on the namespace. Multi-tenant clusters without quotas are one bad YAML file away from a bad day.

What Are Resource Quotas?

A ResourceQuota is a namespaced Kubernetes object that limits the aggregate resource consumption — CPU, memory, storage, and even object counts (Pods, Services, ConfigMaps, etc.) — within a namespace. It’s the primary mechanism for enforcing fair resource sharing in multi-tenant clusters, where multiple teams or applications share the same underlying infrastructure.

Resource quotas work alongside, not instead of, per-container resource requests/limits — quotas cap the namespace total, while LimitRange (covered below) governs individual container defaults and bounds.

How Resource Quotas Are Enforced

When a ResourceQuota exists in a namespace, the API server’s admission controller checks every new object creation/update against the quota before it’s persisted to etcd. If accepting the request would exceed any quota, the request is rejected outright — the pod is never even created, not created-then-evicted.

This matters: quota enforcement happens at admission time, synchronously, as part of the request itself.

Basic Compute Resource Quota

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-compute-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
kubectl apply -f team-a-quota.yaml
kubectl describe resourcequota team-a-compute-quota -n team-a
Name:            team-a-compute-quota
Namespace:       team-a
Resource         Used   Hard
--------         ----   ----
limits.cpu       0      20
limits.memory    0      40Gi
requests.cpu     0      10
requests.memory  0      20Gi

A critical detail: once a ResourceQuota exists in a namespace, every pod created there must specify resource requests/limits for the resources the quota governs — pods without them are rejected outright, since the admission controller has no value to count against the quota.

kubectl apply -f pod-without-limits.yaml
Error from server (Forbidden): error when creating "pod-without-limits.yaml":
pods "test-pod" is forbidden: failed quota: team-a-compute-quota:
must specify limits.cpu for: app; limits.memory for: app; requests.cpu for: app; requests.memory for: app

This is by design — it forces good hygiene across every team sharing the cluster.

Object Count Quotas

Beyond compute, you can cap how many of each object type a namespace can create:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-object-quota
  namespace: team-a
spec:
  hard:
    pods: "50"
    services: "10"
    services.loadbalancers: "2"
    persistentvolumeclaims: "20"
    configmaps: "30"
    secrets: "30"
    count/deployments.apps: "15"
    count/jobs.batch: "20"

services.loadbalancers: "2" is particularly worth setting explicitly — LoadBalancer services provision real cloud infrastructure (and incur real cost) each time one is created, so capping this prevents runaway cloud spend from a namespace spinning up more external load balancers than intended.

Storage Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-storage-quota
  namespace: team-a
spec:
  hard:
    requests.storage: 500Gi
    persistentvolumeclaims: "20"
    gold.storageclass.storage.k8s.io/requests.storage: 100Gi
    standard.storageclass.storage.k8s.io/requests.storage: 400Gi

Storage-class-scoped quotas (the <class-name>.storageclass.storage.k8s.io/requests.storage syntax) let you enforce different limits for expensive, fast storage classes (gold) versus cheaper standard ones — useful when a team should have generous access to standard storage but tightly limited access to premium tiers.

Quota Scopes: Applying Rules to Subsets of Pods

Scopes let you apply a quota only to pods matching certain criteria — priority class, QoS tier, or terminating state:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: high-priority-quota
  namespace: team-a
spec:
  hard:
    pods: "10"
  scopes:
  - PriorityClass
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values: ["high-priority"]

This caps how many high-priority-classed pods can run in the namespace — separate from the general pod count quota — preventing a team from marking everything as high-priority to bypass normal scheduling fairness.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: besteffort-quota
  namespace: team-a
spec:
  hard:
    pods: "5"
  scopes:
  - BestEffort

This limits pods with no resource requests/limits at all (BestEffort QoS class) — useful for namespaces that should mostly run properly-sized workloads, with only a small allowance for ad-hoc debugging pods.

LimitRange: Setting Defaults and Bounds Per Container

ResourceQuota caps namespace totals; LimitRange governs individual container-level defaults, minimums, and maximums — the two work together.

apiVersion: v1
kind: LimitRange
metadata:
  name: team-a-limits
  namespace: team-a
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    max:
      cpu: "2"
      memory: "4Gi"
    min:
      cpu: "50m"
      memory: "64Mi"
  - type: PersistentVolumeClaim
    max:
      storage: 50Gi
    min:
      storage: 1Gi

With this in place, a container spec that omits resource requests/limits entirely automatically gets default/defaultRequest applied — which also satisfies the ResourceQuota requirement mentioned earlier, since the values now exist even if the developer didn’t specify them.

kubectl apply -f team-a-limitrange.yaml
kubectl describe limitrange team-a-limits -n team-a
Name:       team-a-limits
Namespace:  team-a
Type        Resource  Min    Max   Default Request  Default Limit
----        --------  ---    ---   ---------------  -------------
Container   cpu       50m    2     250m             500m
Container   memory    64Mi   4Gi   256Mi            512Mi

Multi-Tenant Production Pattern

A typical multi-team production setup combines both objects per namespace:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: production-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    requests.storage: 200Gi
    pods: "100"
    services.loadbalancers: "1"
    count/deployments.apps: "25"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: production-limits
  namespace: team-payments
spec:
  limits:
  - type: Container
    default:
      cpu: "1"
      memory: "1Gi"
    defaultRequest:
      cpu: "500m"
      memory: "512Mi"
    max:
      cpu: "4"
      memory: "8Gi"

Applied together, this gives the team-payments namespace generous but bounded capacity, sensible per-container defaults so developers don’t have to think about resource specs for every single Deployment, and a hard ceiling that protects the rest of the cluster from a runaway workload.

Automating Quota Provisioning for New Namespaces

In larger orgs, quotas should be provisioned automatically whenever a new team namespace is created, rather than manually and inconsistently. A common pattern with GitOps (ArgoCD/Flux) is a namespace template that always ships with a paired ResourceQuota and LimitRange:

# namespace-template/kustomization.yaml
resources:
  - namespace.yaml
  - resourcequota.yaml
  - limitrange.yaml
kubectl kustomize namespace-template/ | kubectl apply -f -

This guarantees no namespace ever exists without baseline guardrails, regardless of who created it or how.

Monitoring Quota Usage

Export quota utilization to Prometheus via kube-state-metrics and alert before teams hit hard limits and start seeing rejected deployments:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: resourcequota-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus
spec:
  groups:
  - name: resourcequota
    rules:
    - alert: NamespaceQuotaNearLimit
      expr: |
        kube_resourcequota{type="used"} / kube_resourcequota{type="hard"} > 0.85
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: "Namespace {{ $labels.namespace }} quota for {{ $labels.resource }} near limit"
        description: "{{ $labels.resource }} usage is above 85% of quota in {{ $labels.namespace }}."

Troubleshooting

# Check current usage vs hard limits
kubectl describe resourcequota -n <namespace>

# Find why a pod creation was rejected
kubectl apply -f pod.yaml
# read the Forbidden error message carefully — it names the exact resource that exceeded quota

# List all quotas across the cluster
kubectl get resourcequota --all-namespaces

RBAC and ResourceQuota: Who Can Change the Ceiling

A resource quota is only a meaningful safeguard if teams can’t simply raise it themselves. In most multi-tenant setups, namespace admins retain the ability to create workloads within their namespace but should not be able to edit the ResourceQuota object itself — that’s a platform-team responsibility. Enforce this with a Role scoped to exclude quota modification:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: team-namespace-admin
  namespace: team-payments
rules:
- apiGroups: ["apps", ""]
  resources: ["deployments", "pods", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: [""]
  resources: ["resourcequotas", "limitranges"]
  verbs: ["get", "list", "watch"]

Notice resourcequotas and limitranges are explicitly limited to read-only verbs here — the team can see their current allocation and usage, but modifying the ceiling itself requires a separate, more privileged role held only by platform administrators.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: team-payments-admin-binding
  namespace: team-payments
subjects:
- kind: Group
  name: team-payments-engineers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: team-namespace-admin
  apiGroup: rbac.authorization.k8s.io

This separation — teams can consume up to the quota, only platform admins can change the quota — is what actually makes a ResourceQuota an enforceable governance mechanism rather than a suggestion.

Requesting a Quota Increase: A Practical Workflow

Rather than ad hoc Slack messages asking for more capacity, a GitOps-based quota request process keeps changes auditable and reviewable like any other infrastructure change:

git diff namespaces/team-payments/resourcequota.yaml
- requests.cpu: "20"
+ requests.cpu: "30"

Submitted as a pull request against the platform team’s namespace-config repository, this gives reviewers visibility into exactly what’s changing, why (via the PR description), and a clear audit trail for capacity planning conversations later — much better than remembering that someone manually ran kubectl edit resourcequota during an incident six months ago.

Common Mistakes

  • Setting a ResourceQuota without a matching LimitRange — every developer is then forced to manually specify requests/limits on every container, with no sensible defaults, leading to inconsistent or copy-pasted values.
  • Quota limits set arbitrarily without basing them on actual historical usage — either too tight (constant rejected deployments) or too loose (quota provides no real protection).
  • Forgetting services.loadbalancers quota specifically — this is the one that translates directly into unexpected cloud infrastructure costs.
  • Not monitoring quota utilization, so teams discover they’ve hit a limit only when a deployment fails during an incident.
  • Applying identical quotas to every namespace regardless of actual team size or workload — quotas should reflect real capacity planning, not a one-size-fits-all template.
  • No object count quotas, allowing namespace sprawl (hundreds of ConfigMaps, Secrets, or Services) that has nothing to do with compute but still degrades API server performance at scale.

Summary

ResourceQuotas are the primary safeguard against any single namespace consuming more than its fair share of a shared cluster’s compute, storage, or object count — enforced synchronously at admission time, not after the fact. Pair them with LimitRange for sensible per-container defaults, apply them consistently via GitOps templates for every new namespace, and monitor utilization with Prometheus so teams see capacity pressure coming before it blocks a deployment. In any multi-tenant cluster, this isn’t optional hardening — it’s baseline hygiene.

References

Total
1
Shares
Leave a Reply
Previous Post
How to Set Up CI/CD with Jenkins and Kubernetes

How to Set Up CI/CD with Jenkins and Kubernetes

Next Post
How to Set Up Prometheus Alerting in Kubernetes

How to Set Up Prometheus Alerting in Kubernetes

Related Posts