Appearance
What happens when Kubernetes components fail?
If you look at Kubernetes strictly as a clustered database with a bunch of reconciliation loops bolted onto it, its failure domains start to make a lot of sense. The system is radically decoupled. When a process crashes, it rarely takes down the whole ship—in fact, the data plane (your running apps) usually survives entirely disconnected from the control plane.
When failures occur, the impact depends heavily on whether you lost state, lost compute, or simply lost the ability to issue commands temporarily.
1. The API Server Goes Down
The kube-apiserver is the front door. It’s the central HTTP gateway that every other component talks to.
If this process crashes—or the VM hosting it completely dies—you lose the ability to manage the cluster. You can't deploy new definitions, scale deployments, or delete pods.
However, because of the decoupled architecture, the data plane surviving the outage is completely fine. The kubelets on your worker nodes will keep diligently running the container processes they were already assigned, and the kube-proxy will keep routing traffic. The apps stay up, but the cluster is effectively "frozen in time."
TIP
This is why you run multiple API servers behind a highly-available (HA) load balancer. If one API server dies, the kubelets seamlessly connect to the next available instance.
2. Losing the etcd Brain
etcd is the heart of the cluster—it's the only place where true cluster state actually lives. Every other component is effectively stateless.
If your etcd ring degrades or the data becomes corrupted, the API server can't read or write state, and it will fail. The kubelets will lose touch with the control plane, but again, your existing pods will keep running.
This is the most dangerous failure mode in Kubernetes. Without etcd, there is no cluster.
WARNING
You cannot self-heal from a catastrophic etcd data loss. If the Quorum is lost and unrecoverable, you are reaching for your backup snapshots to manually restore state. Always run etcd on extremely fast, reliable disks (like NVMe or specialized EBS volumes) with automated, off-node snapshotting.
3. Controller Manager & Scheduler Crashes
These background workers are constantly crunching state. The kube-scheduler assigns pods to nodes, and the kube-controller-manager ensures that your ReplicaSets actually have the right number of pods.
If the controller manager dies, your cluster stops actively reconciling. If a worker node burns down during this time, the cluster won't spin up replacement pods because the controller manager isn't there to notice the missing replicas. Similarly, if the scheduler dies, any newly created pods will just sit in a Pending state forever because no one is there to assign them a home.
Luckily, these components are totally stateless. When you run them in HA, they use a built-in leader election lock in etcd. If the active scheduler faults, a standby replica immediately grabs the lock and resumes the workload without skipping a beat.
4. Worker Nodes Disappearing (Network Partitions)
Compute nodes actually execute your workloads. If a physical machine experiences a hardware failure or the network partitions, the kubelet on that node stops sending its heartbeat to the API server.
From the control plane's perspective, the node has simply gone silent. After a configurable timeout (usually about 5 minutes), the node controller will officially mark the node as NotReady. It assumes everything on that machine is unreachable and triggers an eviction, instructing the surrounding workload controllers to spin up replacement pods on healthy nodes.
NOTE
Network partitions can be tricky. If the node is perfectly fine but just can't reach the control plane, the pods are technically still running. When the partition heals, the kubelet reconnects, realizes it was presumed unavailable, and will terminate the old pods to gracefully clear the slate.
5. The Kubelet Faults
The kubelet is the ultimate boss of the local machine. It talks directly to containerd to make sure your Linux containers are actually running.
If the kubelet process crashes due to a software bug, it stops reporting host metrics and pod statuses to the API server. Just like a network partition, the control plane will eventually declare the node dead and reschedule the workloads.
Because the kubelet is usually managed directly by systemd on the Linux host machine, the most reliable mitigation is natively configuring systemd to automatically restart the process if it exits with an error code.
6. Node Starvation (OOMs and Evictions)
This is an incredibly common, nasty failure at the workload level. Applications misbehave, leak memory, and consume more compute than the machine actually has.
To prevent the entire Linux host from locking up, the kubelet aggressively monitors memory and disk pressure. If pressure hits critical thresholds, the kubelet initiates Node-pressure eviction. It overrides standard grace periods and immediately terminates running pods to reclaim memory.
If the memory spike is too rapid for the kubelet to react, the Linux kernel's Out-Of-Memory (oom_killer) process intervenes.
TIP
Kubernetes utilizes the oom_killer by assigning Linux oom_score_adj values based on your pod's Quality of Service (QoS). Containers in a highly-constrained Guaranteed QoS class get a highly protective score of -997, while unconstrained BestEffort pods get a 1000. When memory runs out, the kernel will terminate the unconstrained BestEffort workloads first to seamlessly protect your critical infrastructure.