Etcd Disaster Recovery: Safe Restoration Workflows

How should I handle etcd restores when API servers are active?

You must never attempt to restore an etcd cluster while any Kubernetes API servers (kube-apiserver) are running.

Because etcd acts as the absolute source of truth for a Kubernetes cluster, restoring it while the kube-apiserver is active can lead to severe data corruption and state mismatches.

The API server maintains continuous, long-lived connections to etcd to populate its in-memory caches and process watch events. If the underlying database is suddenly rolled back to a previous snapshot or replaced, the active API servers will be completely unaware of the state change. They will continue to write conflicting data based on stale memory, or serve fundamentally broken state to your controllers, nodes, and end-users.

To safely perform a restore, you must strictly follow a sequence that completely isolates the database before introducing the new state.

---

The Strict 5-Step Operational Restore Sequence

1. Stop All API Servers

Before initiating a restore operation, you must explicitly stop all kube-apiserver instances across your entire control plane. This severs the connection between the cluster's control loops and the database, preventing any in-flight requests from mutating the state during the crucial restore window.

• (If running API servers as static Pods, briefly move their manifests out of /etc/kubernetes/manifests to force the kubelet to terminate them).

2. Restore the etcd State

With the API servers successfully stopped, use the etcdutl command-line tool to restore your cluster from a previous snapshot file (e.g., snapshot.db).

You must specify a new directory for the restored cluster using the --data-dir flag.

# This command will automatically create the new directory and populate it
etcdutl --data-dir <new-data-dir-location> snapshot restore snapshot.db

3. Reconfigure the API Servers (If Applicable)

If the architecture of the restored etcd cluster differs from the previous one—specifically, if the access URLs or IP addresses have changed—you must reconfigure the Kubernetes API servers before starting them back up.

You do this by updating the --etcd-servers=$NEW_ETCD_CLUSTER command-line flag on the kube-apiserver manifests. If you are using an external load balancer in front of the etcd cluster, ensure the backend targets are updated to point mapped to the new, restored members.

4. Restart the API Servers

Once the etcd cluster is successfully restored, running, and visibly healthy, you can safely restart all kube-apiserver instances.

• (Move the static Pod manifests back into /etc/kubernetes/manifests).

The API servers will boot, connect to the newly restored etcd database, and rebuild their internal caches entirely based on this new, clean source of truth.

5. Flush Component Caches

This is the most frequently missed step. Even though the API server is now serving the correct state, other automated components in the cluster may still be operating on cached data from before the disaster event.

The Kubernetes project strongly recommends bouncing the local caches of all primary controllers. You must restart the following daemon services across your cluster:

• kube-scheduler
• kube-controller-manager
• kubelet (on all worker nodes)

This ensures that all control loops and node agents drop any stale state they were holding in memory, forcing them to synchronize cleanly against the newly restored database.