Recover Kubernetes Cluster from etcd Snapshot

Kubernetes clusters depend entirely on etcd as their source of truth for all cluster state information. When disaster strikes—whether through hardware failure, accidental deletion, corruption, or malicious activity—etcd snapshot restoration becomes your most reliable path to recovery. This comprehensive guide walks through every aspect of etcd-based Kubernetes recovery, from preparation through validation and into production hardening.

Understanding etcd’s Critical Role in Kubernetes

etcd serves as Kubernetes’ distributed key-value store, maintaining all objects including pods, services, deployments, secrets, configmaps, and cluster configuration. A single etcd failure can render an entire cluster inoperable, making regular snapshots mission-critical for any production environment.

Key characteristics of etcd snapshots make them ideal for recovery:

• Point-in-time consistency: Snapshots capture the exact cluster state at backup time
• Single file portability: One .db file contains complete cluster state
• Version compatibility: Supports restoration across compatible etcd versions
• Lightweight: Typically 100MB-2GB for production clusters

Preparation: Building Your Recovery Foundation

Successful restoration begins with proper preparation. Before disaster hits, ensure these prerequisites are met:

Validating Snapshot Quality

Before any restoration attempt, rigorously validate your snapshot:

ETCDCTL_API=3 etcdctl snapshot status snapshot.db \  --write-out=table \  --cacert=/etc/kubernetes/pki/etcd/ca.crt \  --cert=/etc/kubernetes/pki/etcd/server.crt \  --key=/etc/kubernetes/pki/etcd/server.key

Look for these critical health indicators:

• Hash verification: No corruption detected
• Revision numbers: Sequential and increasing
• Key count: Matches expected cluster size
• Size: Reasonable for your workload

Step-by-Step Restoration Process

Phase 1: Isolate the Control Plane

The first step prevents kube-apiserver and other components from interfering with etcd operations:

1. Move static pod manifests:
   mv /etc/kubernetes/manifests /etc/kubernetes/manifests.backup
2. Wait for drain:
   sleep 15
3. Stop etcd service:
   systemctl stop etcd

Phase 2: Preserve Current State

Always backup the existing etcd data directory before restoration:

CURRENT_TIME=$(date +%Y%m%d-%H%M%S)BACKUP_DIR="/var/lib/etcd.backup-$CURRENT_TIME"# Create timestamped backupmv /var/lib/etcd "$BACKUP_DIR"# Verify backup createdls -la "$BACKUP_DIR"

Phase 3: Execute Snapshot Restoration

Use etcdctl to restore the snapshot to a fresh data directory:

ETCDCTL_API=3 etcdctl snapshot restore snapshot.db \  --data-dir=/var/lib/etcd-restored \  --name=control-plane-1 \  --initial-cluster=control-plane-1=https://192.168.1.10:2380 \  --initial-cluster-token=etcd-cluster-restored-$(date +%s) \  --initial-advertise-peer-urls=https://192.168.1.10:2380 \  --cacert=/etc/kubernetes/pki/etcd/ca.crt \  --cert=/etc/kubernetes/pki/etcd/server.crt \  --key=/etc/kubernetes/pki/etcd/server.key

Key parameters explained:

• --data-dir: Target directory for restored data
• --name: etcd member name (must match kubeadm configuration)
• --initial-cluster: Complete cluster topology
• --initial-cluster-token: Creates new cluster identity

Phase 4: Secure and Activate

1. Replace data directory:
   rm -rf /var/lib/etcd && mv /var/lib/etcd-restored /var/lib/etcd
2. Set ownership:
   chown -R etcd:etcd /var/lib/etcd && chmod -R 700 /var/lib/etcd
3. Restart etcd:
   systemctl start etcd

Complete Automation Script

Here’s a production-ready restoration script incorporating all best practices:

#!/bin/bash# production-etcd-restore.shset -euo pipefailSNAPSHOT_FILE=${1:?"Usage: $0 <snapshot.db>"}CONTROL_PLANE_IP=${2:-"192.168.1.10"}ETCD_NAME=${3:-"control-plane-1"}# Color outputgreen() { echo -e "\033[32m$*\033[0m"; }red() { echo -e "\033[31m$*\033[0m"; exit 1; }backup_current_etcd() {  local backup_dir="/var/lib/etcd.backup-$(date +%Y%m%d-%H%M%S)"  green "Backing up current etcd to $backup_dir"  systemctl stop etcd  mv /var/lib/etcd "$backup_dir"  echo "$backup_dir" > /root/last_etcd_backup}restore_snapshot() {  green "Restoring snapshot: $SNAPSHOT_FILE"  ETCDCTL_API=3 etcdctl snapshot restore "$SNAPSHOT_FILE" \    --data-dir=/var/lib/etcd \    --name="$ETCD_NAME" \    --initial-cluster="$ETCD_NAME=https://$CONTROL_PLANE_IP:2380" \    --initial-cluster-token=etcd-cluster-restored-$(date +%s) \    --initial-advertise-peer-urls=https://$CONTROL_PLANE_IP:2380 \    --cacert=/etc/kubernetes/pki/etcd/ca.crt \    --cert=/etc/kubernetes/pki/etcd/server.crt \    --key=/etc/kubernetes/pki/etcd/server.key || red "Snapshot restore failed"    chown -R etcd:etcd /var/lib/etcd  chmod -R 700 /var/lib/etcd}verify_etcd_health() {  green "Verifying etcd health..."  if ETCDCTL_API=3 etcdctl endpoint health \    --endpoints=https://127.0.0.1:2379 \    --cacert=/etc/kubernetes/pki/etcd/ca.crt \    --cert=/etc/kubernetes/pki/etcd/server.crt \    --key=/etc/kubernetes/pki/etcd/server.key >/dev/null; then    green "✓ etcd healthy"  else    red "✗ etcd health check failed"  fi}restart_control_plane() {  green "Restarting control plane..."  mv /etc/kubernetes/manifests.backup /etc/kubernetes/manifests    # Wait for API server  for i in {1..60}; do    if kubectl cluster-info &>/dev/null; then      green "✓ API server ready"      return 0    fi    sleep 5  done  red "API server failed to start"}main() {  # Phase 1: Isolate  green "=== Phase 1: Isolating control plane ==="  mv /etc/kubernetes/manifests /etc/kubernetes/manifests.backup  sleep 15    # Phase 2: Backup  backup_current_etcd    # Phase 3: Restore  green "=== Phase 4: Restoring snapshot ==="  restore_snapshot  systemctl start etcd  sleep 10  verify_etcd_health    # Phase 5: Restart  restart_control_plane    green "=== VALIDATION ==="  kubectl get nodes  kubectl get pods -n kube-system  green "🎉 CLUSTER RESTORED SUCCESSFULLY!"}main "$@"

Post-Restoration Validation Checklist

Multi-Master Cluster Considerations

For HA clusters with stacked etcd topology:

1. Sequential restoration: Restore master-1 first, verify health
2. Member re-addition: Other masters join restored cluster
3. Quorum verification: Majority of etcd members operational

# On master-1 (restored)ETCDCTL_API=3 etcdctl member add master-2 \  --peer-urls=https://192.168.1.11:2380# On master-2ETCDCTL_API=3 etcdctl member add master-3 \  --peer-urls=https://192.168.1.12:2380

Production Hardening Recommendations

Automated Backup Pipeline

#!/bin/bash# Daily etcd backup to S3SNAPSHOT_FILE="/backups/etcd-$(date +%Y%m%d-%H%M).db"ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \  --cacert=/etc/kubernetes/pki/etcd/ca.crt \  --cert=/etc/kubernetes/pki/etcd/server.crt \  --key=/etc/kubernetes/pki/etcd/server.key \  snapshot save "$SNAPSHOT_FILE"aws s3 cp "$SNAPSHOT_FILE" s3://k8s-backups/etcd-snapshots/aws s3 cp "$SNAPSHOT_FILE" s3://k8s-backups/etcd-snapshot-latest.db --metadata-directive REPLACE

Disaster Recovery Runbook

# Emergency rollbacksystemctl stop etcdrm -rf /var/lib/etcdmv /root/last_etcd_backup /var/lib/etcdsystemctl start etcd

Common Pitfalls and Solutions

Frequently Asked Questions

Can I restore to a different Kubernetes version?

etcd snapshots are generally backward-compatible within major.minor versions. Always test version compatibility in staging first. Kubernetes 1.28 can typically restore etcd snapshots from 1.25+.

What happens to running workloads during restoration?

Workloads continue running on worker nodes using cached API server state. Critical applications should use external databases for persistence beyond etcd.

How frequently should I take snapshots?

Production recommendation: Every 30 minutes with 24-hour retention. Balance RPO requirements against storage costs.

External etcd vs stacked topology?

Both use identical restoration procedures. External etcd requires restoring all etcd nodes; stacked etcd restores via kubeadm static pods.

Monitoring and Alerting

Implement these observability patterns:

• etcd metrics: etcd_server_has_leader, etcd_cluster_is_available
• Backup success: Alert on failed snapshot creation
• Recovery time: Track restoration duration
• Post-restore validation: Automated cluster health checks