etcd backup and restore
Author: Fabrice JAMMES (LinkedIn).
Objective
- Read the keys of the Kubernetes objects in etcd, with
etcdctland the client certificates of etcd - Back up etcd with a snapshot, and check the snapshot
- Restore the snapshot, and get back a deleted object
etcd holds the whole state of the cluster: all the objects, including the Secrets. A backup of etcd is the backup of the cluster, and a sensitive file. The OpenShift version of this lab is etcd administration.
Official documentation, available in the exam: Operating etcd clusters for Kubernetes.
Prerequisites
The lab restores etcd, so the API server restarts: use a kind cluster of your own, never a shared one.
$CP is the control plane node: kind runs it as a docker container, and etcd runs in it as a static pod. The image of the etcd pod has no shell: work on the node, as in the exam, where etcdctl and etcdutl are installed on the control plane node.
Install them on the kind node, in the version of the etcd of the cluster:
Exercise 1: the keys of the Kubernetes objects
- Which address does etcd listen on? Which certificates does a client need?
- List the keys of the
defaultnamespace in etcd. Under which prefix does Kubernetes store its objects?
Tip
Read the command line of the etcd static pod: --listen-client-urls, --trusted-ca-file, --cert-file, --key-file.
Exercise 2: back up etcd
- Create a ConfigMap
before-backupin thedefaultnamespace. - Save a snapshot of etcd in
/var/lib/etcd-backup/snapshot.dbon the control plane node, withetcdctl. - Check the snapshot with
etcdutl: its hash, its revision, its number of keys. - Copy the snapshot out of the node.
Exercise 3: restore the snapshot
- Delete the ConfigMap
before-backup, and create a ConfigMapafter-backup. - Stop the API server.
- Restore the snapshot in a new data directory,
/var/lib/etcd-restore, withetcdutl, and make theetcdstatic pod use this directory. - Check in etcd which ConfigMaps exist, then start the API server again.
Warning
Very important for the CKS exam: save a copy of the manifests outside /etc/kubernetes/manifests, e.g. in /root. The kubelet runs every file of this directory as a static pod: a backup copy there would start a second etcd.
Questions
- Why restore in a new directory, rather than in
/var/lib/etcd? - Who can read the snapshot of etcd? What does it contain?
- How does the restore work on a cluster with three control plane nodes?
Clean up
On the CKS training server, your cluster keeps the restored etcd, in /var/lib/etcd-restore: it is harmless for the next labs.