Kubernetes Cluster Failure Troubleshooting Workflow

This systematic workflow helps you diagnose and resolve Kubernetes cluster failures by moving from broad checks to specific fixes. You’ll start with cluster-level health, then drill into workloads, networking, and storage, using kubectl commands at each stage. No prior debugging experience is needed, but basic familiarity with kubectl and YAML is helpful.

What You’ll Need

• Access to the cluster via kubectl (or another client) with permissions to view nodes, pods, events, and logs.
• The kubeconfig file for the cluster you’re troubleshooting.
• Basic understanding of Kubernetes objects: Pods, Deployments, Services, Nodes, and Namespaces.
• Optional: a terminal with jq for parsing JSON output, and a text editor for YAML files.

Step-by-Step

1. Check cluster-level health. Run kubectl get nodes and look for Ready status. If any node is NotReady, investigate with kubectl describe node <node-name> to see conditions and events. Check kubelet status on the node (e.g., systemctl status kubelet or journalctl -u kubelet).
2. Verify control plane components. If nodes are ready but the API server is unresponsive, check the control plane pods (in kube-system namespace) with kubectl get pods -n kube-system. Look for CrashLoopBackOff or ImagePullBackOff. Describe the failing pod to see events.
3. List all namespaces and workloads. Run kubectl get all --all-namespaces to see the overall state. Identify which workloads are not ready. Use kubectl get pods -n <namespace> to focus.
4. Inspect pod status. For each pod not in Running/Ready, run kubectl describe pod <pod-name> -n <namespace>. Look at the Events section at the bottom for clues like FailedScheduling, BackOff, or Unhealthy.
5. Check pod logs. If the pod is running but crashing, get logs with kubectl logs <pod-name> -n <namespace> --previous (to see the last crashed attempt). For init containers, use kubectl logs <pod-name> -c <init-container-name>.
6. Test service connectivity. If pods are running but the service is not reachable, check endpoints with kubectl get endpoints <service-name> -n <namespace>. If endpoints are empty, check the selector in the service matches the pod labels. Also verify the service type (ClusterIP, NodePort, LoadBalancer) and that the targetPort matches the container port.
7. Check network policies and DNS. If pods can’t talk to each other, check NetworkPolicies with kubectl get networkpolicy -n <namespace>. For DNS issues, test with kubectl exec <pod> -- nslookup kubernetes.default. Check CoreDNS pods in kube-system.
8. Inspect storage issues. If pods are stuck in ContainerCreating, check persistent volume claims with kubectl get pvc -n <namespace> and kubectl describe pvc <name>. Ensure the PV is bound and the storage class exists.
9. Review resource limits and quotas. If pods are pending, check if they are requesting more CPU/memory than available. Use kubectl describe node to see allocatable resources. Also check ResourceQuotas in the namespace with kubectl get resourcequota.
10. Check for taints and tolerations. If pods are unschedulable, see if nodes have taints that the pod doesn’t tolerate: kubectl describe node | grep -A5 Taints. Add tolerations if needed.
11. Look at cluster events. Use kubectl get events --sort-by='.lastTimestamp' to see recent events across the cluster. This often reveals the root cause even if the pod looks fine now.
12. Check API server and etcd. If the API server is slow or unresponsive, check its logs (if you have access) and etcd health with kubectl get --raw='/healthz' or etcdctl. Look for disk space issues on control plane nodes.
13. Verify kube-proxy and CNI. If services are not working, check kube-proxy pods and the CNI plugin (e.g., Calico, Flannel). Look at the CNI daemonset status and logs.
14. Apply fixes incrementally. Once you identify the cause, make changes (e.g., fix YAML, delete stuck pods, scale up nodes) and re-check. Use kubectl rollout restart deployment <name> if needed.
15. Document and monitor. After resolution, note the cause and fix. Set up monitoring (e.g., Prometheus, Grafana) to catch similar issues early.

Common Mistakes

• Ignoring pod events – Events often contain the exact reason (e.g., FailedScheduling, BackOff). Skipping them wastes time.
• Checking logs without –previous – If a pod is crashing, the current logs may be empty; you need the previous container’s logs.
• Forgetting namespaces – Many commands default to the ‘default’ namespace, but your workload may be elsewhere. Always specify -n.
• Assuming the service selector matches – A typo in labels is a common cause of empty endpoints.
• Overlooking resource limits – A pod might be evicted or stuck if it exceeds memory limits, even if it’s running.
• Not checking node pressure – Disk pressure, memory pressure, or PID pressure on nodes can cause evictions and scheduling failures.

Tips

• Use kubectl get pods -o wide to see which node each pod runs on, which helps spot node-specific issues.
• Set up a kubectl alias like alias k='kubectl' to speed up typing.
• Use kubectl get all -n <namespace> -o wide for a quick overview with IPs and status.
• When a pod is CrashLoopBackOff, check the exit code in the pod description – it often points to the problem (e.g., 137 = OOM killed, 1 = app error).
• For network issues, use a temporary debug pod: kubectl run -it --rm debug --image=nicolaka/netshoot -- /bin/bash.
• Keep a cheat sheet of common kubectl commands and their output meanings.
• If you have access to the cluster’s cloud console, check node health and autoscaling events there.

Troubleshooting

If your initial steps don’t reveal the issue, try these additional approaches:

• Check the kubelet logs on the affected node – You might need SSH access. Look for errors about container runtime, CNI, or image pulls.
• Verify the container runtime – If using containerd or CRI-O, check its status and logs. A misconfigured runtime can cause pods to fail to start.
• Inspect the cluster’s cloud provider integration – If using a managed service (EKS, GKE, AKS), check the provider’s status page and your cloud console for node group issues.
• Check for API rate limiting – If you see 429 errors, you may be hitting API server limits. Reduce the frequency of kubectl calls.
• Use a debugging container – If a pod is stuck in Pending, create a simple pod with the same nodeSelector and see if it schedules. This isolates the issue.
• Review the cluster’s audit logs – If you have access, audit logs can show who or what made changes that broke things.

Frequently Asked Questions

Why is my pod stuck in Pending?

Pending means the scheduler can’t place the pod. Common causes: insufficient CPU/memory, node taints, or a PersistentVolume that can’t be attached. Describe the pod to see scheduler events.

What does CrashLoopBackOff mean?

The container starts, crashes, and Kubernetes restarts it, but it keeps crashing. Check logs with –previous to see the error. Common causes: misconfiguration, missing dependencies, or the app failing on startup.

How do I fix a service that has no endpoints?

Ensure the service selector matches the pod labels. Run kubectl get pods --show-labels to see actual labels, then compare with the service YAML. Also check that the pods are Running and Ready.

Why can’t my pods resolve DNS names?

Check CoreDNS pods in kube-system. If they are running, test with kubectl exec <pod> -- nslookup kubernetes.default. If that fails, check network policies or if the cluster’s DNS config (kubelet’s –cluster-dns) is correct.

What should I do if a node is NotReady?

SSH into the node and check kubelet status and logs. Common fixes: restart kubelet, check disk space, verify the container runtime is healthy, and ensure the node can reach the API server.

How can I prevent cluster failures in the future?

Implement monitoring and alerting (e.g., Prometheus, Grafana), set resource requests/limits, use pod disruption budgets, and regularly test backups and disaster recovery procedures.

Related

• Disney Shows and Movies to Instantly Get You Hooked
• Keyboards for Programming: A Practical Guide
• Wired Headphones Rebound: Wireless Still Leads