How to Debug Pods in Kubernetes

I don’t think there’s a Kubernetes user alive who hasn’t stared at a Pod stuck in CrashLoopBackOff at 2 a.m., wondering what exactly went wrong. Debugging Pods is one of those skills that separates people who merely deploy to Kubernetes from people who actually understand it. In this guide, I’ll walk through a systematic approach to debugging Pods — from basic status checks to deep container-level inspection — with real commands and outputs you can follow along with.

Understanding Pod Lifecycle First

Before debugging anything, it helps to know what a healthy Pod lifecycle looks like. A Pod moves through these phases: Pending (scheduled but not yet running), Running (at least one container is up), Succeeded, Failed, or Unknown. Within a Pod, each container also has its own state: Waiting, Running, or Terminated. Most debugging boils down to figuring out which state a container is stuck in and why.

Step 1: Check Pod Status

Start broad:

kubectl get pods -n myapp

Output:

NAME                      READY   STATUS             RESTARTS   AGE
myapp-6d9f8b7c5d-4k2pl    0/1     CrashLoopBackOff   5          6m
myapp-6d9f8b7c5d-9xqwt    1/1     Running            0          6m

The STATUS column tells you a lot immediately:

• Pending — usually a scheduling problem (resources, node affinity, taints).
• ImagePullBackOff / ErrImagePull — the image name, tag, or registry credentials are wrong.
• CrashLoopBackOff — the container starts and then exits repeatedly.
• OOMKilled — the container exceeded its memory limit.
• Evicted — the node ran out of resources and Kubernetes removed the Pod.

Step 2: Describe the Pod

kubectl describe is the single most useful debugging command in Kubernetes. It shows events, conditions, resource requests, volume mounts, and recent scheduler decisions all in one place.

kubectl describe pod myapp-6d9f8b7c5d-4k2pl -n myapp

Look specifically at the Events section at the bottom:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  6m                 default-scheduler  Successfully assigned myapp/myapp-6d9f8b7c5d-4k2pl to node-2
  Normal   Pulled     6m                 kubelet            Successfully pulled image "myrepo/myapp:1.2.0"
  Normal   Created    6m                 kubelet            Created container myapp
  Normal   Started    6m                 kubelet            Started container myapp
  Warning  BackOff    2m (x8 over 5m)    kubelet            Back-off restarting failed container

This tells us the image pulled fine and the container started, but it’s crashing after startup — so the problem is inside the application, not the scheduling or image layer.

Step 3: Check Logs

Logs are your next stop:

kubectl logs myapp-6d9f8b7c5d-4k2pl -n myapp

If the container has already crashed and restarted, the current logs might be empty or unhelpful. Use --previous to see the logs from the last terminated instance:

kubectl logs myapp-6d9f8b7c5d-4k2pl -n myapp --previous

For multi-container Pods, specify the container:

kubectl logs myapp-6d9f8b7c5d-4k2pl -c sidecar -n myapp

To stream logs live as they happen:

kubectl logs -f myapp-6d9f8b7c5d-4k2pl -n myapp

Step 4: Exec Into the Container

If the container is at least running (even briefly), you can shell into it directly:

kubectl exec -it myapp-6d9f8b7c5d-4k2pl -n myapp -- /bin/sh

From inside, check environment variables, config files, and connectivity:

env | sort
cat /etc/myapp/config.yaml
curl -v http://dependency-service:8080/health

If the container image doesn’t include a shell (common with distroless or scratch-based images), you won’t be able to exec in traditionally — more on that below.

Step 5: Debug with Ephemeral Containers

For minimal images without a shell or debugging tools, Kubernetes supports ephemeral debug containers that attach to a running Pod’s namespace without modifying the Pod spec:

kubectl debug -it myapp-6d9f8b7c5d-4k2pl -n myapp --image=busybox --target=myapp

This drops you into a busybox shell sharing the same network and process namespace as the target container, letting you inspect processes, run netstat, or check /proc even when the app container itself has no debugging tools installed.

Step 6: Inspect Resource Requests and Limits

A huge number of CrashLoopBackOff and OOMKilled issues trace back to memory limits set too low. Check the Pod spec:

kubectl get pod myapp-6d9f8b7c5d-4k2pl -n myapp -o jsonpath='{.spec.containers[0].resources}'

Example output:

{"limits":{"memory":"128Mi"},"requests":{"cpu":"100m","memory":"64Mi"}}

If your app legitimately needs more memory during startup (JIT warmup, large dependency loading, etc.), bump the limit:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

You can confirm an OOM kill specifically by checking the container’s last state:

kubectl get pod myapp-6d9f8b7c5d-4k2pl -n myapp -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

If it prints OOMKilled, you have your answer.

Step 7: Check Pending Pods and Scheduling Problems

If a Pod never leaves Pending, the culprit is almost always scheduling-related. kubectl describe again gives you the answer, usually something like:

Warning  FailedScheduling  30s (x3 over 2m)  default-scheduler  0/5 nodes are available: 
3 Insufficient memory, 2 node(s) had taint {dedicated: gpu}, that the pod didn't tolerate.

This tells you either to lower resource requests, add more nodes, or add a toleration:

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"

Step 8: Check Readiness and Liveness Probes

Sometimes a Pod is technically “Running” but never becomes “Ready,” which removes it from Service endpoints. Check probe configuration and results:

kubectl describe pod myapp-6d9f8b7c5d-4k2pl -n myapp | grep -A 5 Readiness

A common mistake is setting initialDelaySeconds too low for apps with slow startup, causing the probe to fail before the app is actually ready, which then triggers a restart loop.

Step 9: Network Debugging

If the app runs fine but can’t talk to another service, spin up a temporary debug Pod in the same namespace:

kubectl run netshoot --rm -it --image=nicolaka/netshoot -n myapp -- /bin/bash

From there you get a full toolkit — dig, curl, tcpdump, nslookup — to test DNS resolution and connectivity:

nslookup dependency-service.myapp.svc.cluster.local
curl -v http://dependency-service:8080/health

If DNS resolution fails, check NetworkPolicies that might be blocking traffic, or CoreDNS health:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns

Common Mistakes I’ve Seen

• Ignoring --previous logs and assuming there’s nothing to debug because current logs are empty.
• Setting memory limits based on guesswork instead of actually profiling the app under load.
• Forgetting namespaces — running kubectl logs mypod without -n mynamespace and getting a “not found” error.
• Not checking node-level issues — sometimes the Pod is fine, but the node it landed on is under disk pressure. kubectl describe node <name> will reveal this.
• Overlooking image tag mutability — deploying :latest and getting a version you didn’t expect.

Best Practices for Easier Debugging Later

• Always set both readiness and liveness probes with sensible thresholds.
• Log structured JSON so you can grep and parse quickly under pressure.
• Keep resource requests realistic, based on actual measured usage, not defaults copied from a tutorial.
• Use kubectl get events --sort-by=.lastTimestamp -n myapp to see the full timeline of what’s happened in a namespace recently — it’s often faster than describing individual Pods one by one.

Debugging Init Containers

A category of failure I see people miss constantly: the main container never even starts because an init container is stuck or failing. Init containers run to completion, in order, before any regular container in the Pod starts, and a Pod stuck at Init:0/1 or similar is telling you exactly that.

kubectl get pods myapp-6d9f8b7c5d-4k2pl -n myapp

NAME                      READY   STATUS     RESTARTS   AGE
myapp-6d9f8b7c5d-4k2pl    0/1     Init:1/2   3          4m

Logs for a specific init container work the same way as regular containers, just with -c:

kubectl logs myapp-6d9f8b7c5d-4k2pl -c wait-for-db -n myapp

A very common pattern is an init container waiting on a dependency (a database migration job, a config-loading sidecar) that itself never becomes ready — tracing the chain back one hop at a time with describe and logs on each piece usually finds it quickly.

Debugging DNS Resolution Failures Specifically

DNS issues deserve their own callout because they manifest in confusing ways — an app that “can’t connect” to a dependency is often actually failing to resolve its hostname at all, not failing the connection itself. From inside a debug Pod:

kubectl run dnsdebug --rm -it --image=busybox:1.36 -n myapp -- nslookup kubernetes.default

If this fails, the problem is likely CoreDNS itself, not your specific service:

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

If CoreDNS is healthy but resolution of your specific service fails, double check the Service actually has endpoints (kubectl get endpoints myapp-service -n myapp) — an empty endpoints list, usually from a label selector mismatch, produces a Service that resolves in DNS but connects to nothing.

Debugging Pods That Are Running But Not Ready

A subtly different failure mode from CrashLoopBackOff: a Pod shows Running but 0/1 or partial readiness, meaning the container process is alive but failing its readiness probe. This Pod won’t receive traffic from its Service, but it also won’t restart the way a crashing container would, so it can sit in this state indefinitely without obvious alarm bells.

kubectl describe pod myapp-6d9f8b7c5d-4k2pl -n myapp | grep -A 10 "Readiness"

Common causes: the readiness endpoint depends on a downstream dependency (database, cache) that’s itself unavailable, or the probe’s path/port is simply misconfigured relative to what the app actually serves. I always verify the probe manually from inside the container first, before assuming the app logic itself is broken:

kubectl exec -it myapp-6d9f8b7c5d-4k2pl -n myapp -- curl -v http://localhost:8080/healthz

If that succeeds but the readiness probe still reports failing, double-check the probe’s configured port and path against the container’s actual listening port — a surprisingly common typo.

Debugging Node-Level Problems Affecting Multiple Pods

If several unrelated Pods on the same node start failing simultaneously, stop looking at individual Pods and check the node itself:

kubectl describe node ip-192-168-45-12.ec2.internal

Look at the Conditions section for MemoryPressure, DiskPressure, or PIDPressure set to True — any of these will cause the kubelet to start evicting Pods to relieve resource pressure, and the resulting Pod-level messages (Evicted, generic scheduling failures) can look like an application problem when the real cause is node exhaustion.

kubectl get pods -A --field-selector status.phase=Failed

This surfaces evicted Pods cluster-wide, which is often the fastest way to confirm a node-level rather than app-level root cause.

A Repeatable Debugging Checklist

When I’m handed an unfamiliar cluster with a failing Pod, I run through the same sequence every time rather than guessing:

1. kubectl get pods -n <namespace> — what’s the status?
2. kubectl describe pod <pod> — what do the Events say?
3. kubectl logs <pod> --previous — what did the app say before it died?
4. Check resource limits and the lastState.terminated.reason for OOM kills.
5. Check readiness/liveness probe configuration against actual app behavior.
6. If networking is suspected, spin up a netshoot debug Pod and test DNS and connectivity directly.
7. If the node itself is suspect, kubectl describe node for pressure conditions.

Working this list top to bottom, in order, catches the overwhelming majority of Pod failures well before needing anything more exotic.

Summary

Debugging Pods in Kubernetes follows a fairly consistent funnel: check status, describe for events, check logs (including --previous), exec or attach an ephemeral debug container if you need to go deeper, and then look at resource limits, scheduling constraints, and networking as the usual suspects. Once you’ve internalized this flow, most Pod failures become a five-minute diagnosis instead of a stressful mystery.

References

• Debugging Pods — Kubernetes documentation
• Debugging with ephemeral containers
• Kubernetes troubleshooting clusters
• CNCF Kubernetes project