Backing Up etcd

This guide covers the essential operations for managing the etcd key-value database in a Kubernetes cluster. It walks through verifying the etcd version, retrieving the member list, performing backups (both live snapshot-based and offline file-based), inspecting snapshot integrity, and restoring etcd from a snapshot. The instructions assume etcd is deployed as a static pod on the control plane using Kubernetes v3.x PKI certificates.

Prerequisites

In all KodeKlode Kubernetes hands-on labs, the etcd key-value database is deployed as a static pod on the master node. The version used is v3.

Before performing any operations, verify that etcdctl is running on API version 3.x:

etcdctl version

Example output:

controlplane ~ ➜  etcdctl version
etcdctl version: 3.5.16
API version: 3.5

Get the Member List

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list

Backing Up etcd

Using etcdctl (Snapshot-Based Backup)

To take a snapshot from a running etcd server:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-snapshot.db

Required options:

Using etcdutl (File-Based Backup)

For an offline, file-level backup of the data directory:

etcdutl backup \
  --data-dir /var/lib/etcd \
  --backup-dir /backup/etcd-backup

This copies the etcd backend database and WAL files to the target location.

Checking Snapshot Status

Inspect the metadata of a snapshot file to verify integrity before restore:

etcdctl snapshot status /backup/etcd-snapshot.db --write-out=table

This displays details such as size, revision, hash, and total key count.

Restoring etcd

Using etcdctl (Snapshot Restore)

To restore a snapshot to a new data directory:

ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --name=master \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --data-dir /var/lib/etcd-from-backup \
  --initial-cluster=master=https://127.0.0.1:2380 \
  --initial-cluster-token etcd-cluster-1 \
  --initial-advertise-peer-urls=https://127.0.0.1:2380 \
  snapshot restore /opt/snapshot-pre-boot.db

Using etcdutl (File-Based Restore)

To restore a backup made with etcdutl backup, copy the backup contents back into /var/lib/etcd and restart etcd.

Updating the etcd Static Pod Manifest

After restoring to a new data directory, update /etc/kubernetes/manifests/etcd.yaml:

1. Update --data-dir to the new target location:

   --data-dir=/var/lib/etcd-from-backup

2. Update --initial-cluster-token in the etcd command array:

   --initial-cluster-token=etcd-cluster-1

3. Update volumes and volumeMounts to point to the new path:

   volumeMounts:
     - mountPath: /var/lib/etcd-from-backup
       name: etcd-data
     - mountPath: /etc/kubernetes/pki/etcd
       name: etcd-certs
   hostNetwork: true
   priorityClassName: system-cluster-critical
   volumes:
     - hostPath:
         path: /var/lib/etcd-from-backup
         type: DirectoryOrCreate
       name: etcd-data
     - hostPath:
         path: /etc/kubernetes/pki/etcd
         type: DirectoryOrCreate
       name: etcd-certs

Key Takeaways

• etcdctl snapshot save - Creates a .db snapshot from a live etcd cluster.
• etcdctl snapshot status - Provides metadata about a snapshot file for integrity verification.
• etcdctl snapshot restore - Restores a .db snapshot file to a new data directory.
• etcdutl backup - Performs a raw file-level copy of etcd’s data and WAL files without requiring etcd to be running.

Reference

etcd v3.5 Recovery Guide

• How-To
• Technology

• Kubernetes
• SysAdmin