Kubernetes5 Aug 2026·13 min read

CKA Troubleshooting Labs: 6 Real Exam-Style Cluster Failures

Srinivasa Rao Maganti

Cloud Architect & Lead Trainer, CloudTechTrainings

#CKA#Kubernetes#Troubleshooting#kubectl#DevOps#Certification

The CKA is the one major cloud certification with no multiple choice at all — you get a broken or incomplete cluster and a terminal, and you fix it, live, against the clock. Troubleshooting is also the single largest domain on the exam at 30% weight, ahead of Cluster Architecture (25%), Services & Networking (20%), Workloads & Scheduling (15%), and Storage (10%). Most candidates who run out of time do not fail because they lack Kubernetes knowledge — they fail because they are slow to go from "something is wrong" to "here is the exact command that fixes it." This page is six labs built to close that gap: real symptoms, the diagnosis commands in the order you would actually run them, the root cause, and the fix.

Info: Exam Reality

The real CKA gives you 15–20 performance-based tasks over 2 hours in a live cluster, open-book (docs.kubernetes.io and kubectl are both allowed), with a 66% pass mark. Every lab below mirrors that format: a broken cluster state, not a concept quiz.

The Diagnosis Order That Saves You Time

Before the labs, one habit matters more than any single command: always read Events last in kubectl describe output before anything else — it is usually the fastest path to the real cause. Work top to bottom:

1kubectl get pods -A -o wide — scan for anything not Running/Ready, and note which node it landed on
2kubectl describe pod <name> -n <ns> — scroll straight to Events at the bottom
3kubectl logs <name> -n <ns> [--previous] [-c <container>] — --previous is essential for a pod that already restarted
4kubectl get events -n <ns> --sort-by=.lastTimestamp — cluster-level events the pod description will not show
5For node-level failures: kubectl describe node <name>, then ssh in and check systemctl status kubelet + journalctl -u kubelet

Quick Reference — 6 Failures at a Glance

#	Symptom	First Command	Usual Root Cause
1	CrashLoopBackOff	kubectl describe pod	Liveness probe fires before the app finishes starting
2	Pod stuck Pending	kubectl describe pod	Requested CPU/memory exceeds what any node can offer
3	Node NotReady	kubectl describe node	kubelet stopped — often a cgroup driver mismatch
4	Service unreachable, pods healthy	kubectl get endpoints	Service selector does not match pod labels
5	DNS resolution fails in-pod	kubectl logs -n kube-system	CoreDNS forward loop from a broken node resolv.conf
6	ImagePullBackOff	kubectl describe pod	Missing or wrong-namespace imagePullSecret

1. CrashLoopBackOff — payment-api Won’t Stay Up

payment-api restarts every 30–40 seconds in prod. The container isn’t erroring out — it’s being killed.

bash

$ kubectl get pods -n prod
NAME                          READY   STATUS             RESTARTS      AGE
payment-api-6b47d8f9c-vqz2m   0/1     CrashLoopBackOff   5 (38s ago)   3m12s

$ kubectl describe pod payment-api-6b47d8f9c-vqz2m -n prod
...
Liveness:  http-get http://:8080/healthz delay=2s timeout=1s period=10s #success=1 #failure=3
...
Events:
  Warning  Unhealthy  90s (x3 over 2m)  kubelet  Liveness probe failed: Get "http://10.244.1.15:8080/healthz": dial tcp 10.244.1.15:8080: connect: connection refused
  Normal   Killing    90s               kubelet  Container payment-api failed liveness probe, will be restarted

$ kubectl logs payment-api-6b47d8f9c-vqz2m -n prod --previous
2026-08-04T10:12:03Z  INFO  Loading configuration...
2026-08-04T10:12:03Z  INFO  Connecting to database pool...
2026-08-04T10:12:19Z  INFO  Server listening on :8080

The app is healthy — it just takes ~19 seconds to bind its port while the DB pool warms up. But initialDelaySeconds is 2, so kubelet probes at second 2, gets connection refused, and kills the container before it ever gets a chance. Restart, repeat, forever. This is a race condition, not a crash.

bash

$ kubectl patch deployment payment-api -n prod --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":30}
]'
deployment.apps/payment-api patched

$ kubectl get pods -n prod -w
payment-api-7c9d6f4b5-k8n2p   1/1     Running   0          45s

2. Pod Stuck Pending — batch-worker Never Schedules

batch-worker has sat at 0/1 Pending for six minutes. No CrashLoopBackOff, no ImagePullBackOff — it never even got a node.

bash

$ kubectl get pods -n data
NAME                          READY   STATUS    RESTARTS   AGE
batch-worker-84f9d7c-2xk9p    0/1     Pending   0          6m

$ kubectl describe pod batch-worker-84f9d7c-2xk9p -n data
...
Events:
  Warning  FailedScheduling  90s (x8 over 6m)  default-scheduler  0/3 nodes are available: 3 Insufficient memory.

$ kubectl describe node worker-1 | grep -A6 "Allocated resources"
Allocated resources:
  Resource           Requests      Limits
  --------           --------      ------
  cpu                2400m (60%)   4 (100%)
  memory             3200Mi (91%)  3200Mi (91%)

batch-worker requests 4Gi memory per replica; every node in the cluster only has ~3.5Gi allocatable left. The scheduler is doing exactly its job — it will never place a pod on a node that cannot honor the request, no matter how long you wait.

bash

$ kubectl set resources deployment batch-worker -n data -c worker \
  --requests=memory=512Mi,cpu=250m
deployment.apps/batch-worker resource requirements updated

$ kubectl get pods -n data
NAME                          READY   STATUS    RESTARTS   AGE
batch-worker-84f9d7c-9p2mx    1/1     Running   0          20s

Tip: Second Common Cause of Pending

If node resources look fine, check kubectl describe node | grep Taints next — a node tainted NoSchedule with no matching toleration on the pod produces the identical symptom (0/N nodes available), but for a completely different reason.

3. Node NotReady — worker-2 Drops Out of the Cluster

A cluster-architecture-flavored troubleshooting task: one node has gone NotReady and every pod scheduled on it is now unreachable.

bash

$ kubectl get nodes
NAME       STATUS     ROLES           AGE   VERSION
master-1   Ready      control-plane   40d   v1.31.4
worker-1   Ready      <none>          40d   v1.31.4
worker-2   NotReady   <none>          40d   v1.31.4

$ kubectl describe node worker-2
...
Conditions:
  Type    Status   Reason
  Ready   Unknown  NodeStatusUnknown   Kubelet stopped posting node status.

The control plane has lost contact with kubelet on worker-2. That means SSH to the node itself — kubectl alone cannot fix a dead kubelet.

bash

worker-2$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Active: failed (Result: exit-code)

worker-2$ sudo journalctl -u kubelet -n 20 --no-pager
... "failed to run Kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd""

Classic CKA gotcha: the kubelet config and the container runtime disagree on which cgroup driver to use. It has to match on both sides.

bash

worker-2$ sudo sed -i 's/cgroupDriver: cgroupfs/cgroupDriver: systemd/' /var/lib/kubelet/config.yaml
worker-2$ sudo systemctl daemon-reload
worker-2$ sudo systemctl restart kubelet
worker-2$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Active: active (running)

$ kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
worker-2   Ready    <none>          40d   v1.31.4

4. Service Unreachable — Pods Are Healthy, Traffic Isn’t Arriving

Every frontend pod shows 1/1 Running. The Service in front of them still times out on every request. This is the pattern that sends people straight to network policies — check endpoints first instead.

bash

$ kubectl get pods -n web -l app=frontend
NAME                        READY   STATUS    RESTARTS   AGE
frontend-5d8f7c9b6-abc12    1/1     Running   0          10m
frontend-5d8f7c9b6-def34    1/1     Running   0          10m

$ kubectl get endpoints frontend -n web
NAME       ENDPOINTS   AGE
frontend   <none>      10m

A Service with healthy backing pods but empty Endpoints means exactly one thing: the selector doesn’t match the pod labels. The Service and its pods are living in two different worlds.

bash

$ kubectl get svc frontend -n web -o yaml | grep -A2 selector
  selector:
    app: front-end

$ kubectl get pods -n web --show-labels
NAME                       LABELS
frontend-5d8f7c9b6-abc12   app=frontend,tier=web

app: front-end on the Service, app: frontend on the pods — one hyphen, total outage.

bash

$ kubectl patch svc frontend -n web -p '{"spec":{"selector":{"app":"frontend"}}}'
service/frontend patched

$ kubectl get endpoints frontend -n web
NAME       ENDPOINTS                       AGE
frontend   10.244.1.22:80,10.244.2.15:80   11m

5. DNS Resolution Fails Inside Pods

Pods can reach other pods by IP but not by service name — every internal call using a DNS name is timing out.

bash

$ kubectl run test-dns --image=busybox:1.36 --rm -it --restart=Never \
  -- nslookup backend-svc.prod.svc.cluster.local
;; connection timed out; no servers could be reached

$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS             RESTARTS   AGE
coredns-6f6b9d6fc8-9k2lp   0/1     CrashLoopBackOff   4          5m
coredns-6f6b9d6fc8-p3mzq   0/1     CrashLoopBackOff   4          5m

$ kubectl logs coredns-6f6b9d6fc8-9k2lp -n kube-system
[FATAL] plugin/loop: Loop (127.0.0.1:36769 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting

CoreDNS itself is crashing, and the reason is a forwarding loop: the node’s /etc/resolv.conf points at a local stub resolver that forwards straight back to CoreDNS, which forwards to the stub, forever. CoreDNS’s loop plugin detects this and deliberately crashes rather than spin the cluster network in circles.

bash

worker-1$ cat /etc/resolv.conf
nameserver 127.0.0.53   # systemd-resolved stub — this is the loop

worker-1$ sudo sed -i 's/nameserver 127.0.0.53/nameserver 8.8.8.8/' /etc/resolv.conf

$ kubectl rollout restart deployment coredns -n kube-system
$ kubectl get pods -n kube-system -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-6f6b9d6fc8-r7k2p   1/1     Running   0          20s
coredns-6f6b9d6fc8-t9m4x   1/1     Running   0          20s

6. ImagePullBackOff — A New Deploy Never Starts

inventory-svc was just deployed to staging. It has never run once — straight to ImagePullBackOff.

bash

$ kubectl get pods -n staging
NAME                            READY   STATUS             RESTARTS   AGE
inventory-svc-7c8d9f6b5-k2p9x   0/1     ImagePullBackOff   0          2m

$ kubectl describe pod inventory-svc-7c8d9f6b5-k2p9x -n staging
...
Events:
  Warning  Failed  60s (x4 over 2m)  kubelet  Failed to pull image "registry.internal.example.com/inventory-svc:v2.4.1": rpc error: code = Unauthenticated desc = failed to authorize: 401 Unauthorized

401 Unauthorized, not "not found" — the image exists, but the node has no credentials for this private registry. Secrets (including registry credentials) are namespace-scoped, so a regcred secret that exists in default does nothing for a pod in staging.

bash

$ kubectl create secret docker-registry regcred \
  --docker-server=registry.internal.example.com \
  --docker-username=svc-deploy \
  --docker-password=$REGISTRY_TOKEN \
  --namespace=staging
secret/regcred created

$ kubectl patch deployment inventory-svc -n staging -p \
  '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"regcred"}]}}}}'
deployment.apps/inventory-svc patched

$ kubectl get pods -n staging
NAME                             READY   STATUS    RESTARTS   AGE
inventory-svc-8f6c7d5b4-n4t2q    1/1     Running   0          15s

Tip: Practice Until It’s Muscle Memory

The real exam rewards speed, not just correctness. Rebuild each of these six failures yourself in a local cluster (kind or minikube both work) and re-run the diagnosis commands from memory before you book the exam.

These six patterns cover most of what shows up under the Troubleshooting domain, but they’re drills, not a substitute for cluster-architecture and networking fundamentals.

If you want the full comparison of what Kubernetes gets you over simpler orchestrators first, see Kubernetes vs Docker Swarm. Our 45-day live Kubernetes batch builds these exact debugging instincts through hands-on labs, from Docker fundamentals through production-grade multi-cluster setups.

Put These Labs to the Test

CKA Free Mock Exam — 2 × 30 scenario questions
Instant results, per-question explanations, no account needed
CKAD Free Mock Exam — application developer track
The developer-focused counterpart to CKA
Live Kubernetes Training — CKA & CKAD in 45 days
Hands-on labs on real clusters, not slide decks

Ready to Start Your Cloud Journey?

Live batches Mon–Sat — Azure 9–10 AM IST (started 3 august 2026 — join in progress) · AWS 10:30–11:45 AM IST (starts 31 august 2026). Hands-on labs, exam prep, and community support.

Join Free Demo →WhatsApp Us

Keep Reading

Kubernetes

Kubernetes vs Docker Swarm: Which to Learn in 2026?

An honest comparison for DevOps engineers choosing between K8s and Swarm. Career demand data, feature comparison, and a clear recommendation included.

17 Jun 2026·8 min read

Read →

Career

Is a Cloud Career Right for You? The AZ-900 Beginner’s Guide (2026)

Not a study guide — a decision guide. What cloud computing actually is in plain English, who AZ-900 is really for, an honest salary picture, and the lowest-risk way to test whether this career switch is right for you before spending anything.

5 Aug 2026·9 min read

Read →