Kubelet and Systemd: Process Lifecycle & Cgroups

Why is the Kubelet managed by Systemd, and how do they interact?

While Kubernetes orchestrates containers across a cluster, the Kubernetes node agent (Kubelet) and the Container Runtime (e.g., containerd, CRI-O) are standard Linux binaries that must run directly on the host operating system.

In modern Linux distributions, Systemd serves as the "parent" init system for these critical processes.

---

1. Why Systemd Matters for Kubernetes

Systemd provides three critical functions that Kubernetes relies on for node stability:

1. Process Lifecycle Management: Systemd ensures the Kubelet automatically starts on boot and reliably restarts if it crashes. Since the Kubelet is responsible for keeping all other Pods on the node running, its own uptime is the foundation of the cluster.
2. Graceful Node Shutdown: Kubernetes relies on systemd "inhibitor locks" to securely detect OS shutdown events. This allows the Kubelet to delay the Linux shutdown sequence for a configurable duration (shutdownGracePeriod) so it can safely and gracefully terminate Pods before the power is cut.
3. Cgroup Management: Systemd acts as a kernel control group (cgroup) manager to allocate CPU and Memory limits.

The Cgroup Split-Brain

If the Kubelet uses the legacy cgroupfs driver to manage container resources, but the Linux host itself uses systemd as the init system, you create a dangerous "split-brain" scenario. Two different managers will attempt to control the same kernel resources. Under heavy memory pressure, this conflict inevitably leads to node instability. Best Practice: You must explicitly configure both the Kubelet and your Container Runtime to use the systemd cgroup driver to ensure a single, consistent view of resource allocation.

---

2. The Kubelet Systemd Hierarchy

Unlike the Kubernetes API Server or Scheduler—which typically run natively as static Pods—the Kubelet is a daemon service.

When using kubeadm to bootstrap a node, the kubelet.service is constructed via a strict hierarchy of configuration files. This layered approach allows system administrators to perform upgrades without overwriting custom local settings.

1. Base Unit File: /lib/systemd/system/kubelet.service
   • The standard package instruction shipped with Kubernetes.
2. Kubeadm Drop-in: /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
   • Installed by kubeadm, this configures the Kubelet to look for cluster-specific certificates and points to the dynamic kubeadm-flags.env file.
3. User Overrides: /etc/systemd/system/kubelet.service.d/
   • Administrators place custom overrides here (such as http-proxy.conf to configure corporate proxies). These user files completely override the package defaults.

---

3. Managing and Debugging the Kubelet

Because the Kubelet and the Container Runtime run as raw systemd services, their logs are intercepted by the system journal (journald). They will not appear in standard cluster-level logging pipelines like ElasticSearch or Promtail unless explicitly forwarded.

You must perform node-level debugging using standard Linux tools.

Starting and Reloading

If you modify the Kubelet YAML configuration or add a new systemd drop-in file, you must reload the daemon before it takes effect.

# Reload the systemd manager configuration
sudo systemctl daemon-reload

# Restart the Kubelet service
sudo systemctl restart kubelet

Reading the Journal

To debug a failing node (e.g., node shows NotReady), the journalctl tool is your primary window into the Kubelet's behavior.

• View Real-Time Logs (Tail): Watch logs as they are generated to catch Startup or CrashLoop errors.

  journalctl -u kubelet -f

• Filter by Time: Correlate node failure alerts with specific timestamps.

  journalctl -u kubelet --since "2026-03-10 14:00:00"

• Check the Container Runtime: If the Kubelet logs report Container runtime is down, you must check the CRI logs directly. The runtime must be active before the Kubelet can register the node.

  journalctl -u containerd -f
  # OR
  journalctl -u crio -f