etcd administration

Auteur: Fabrice JAMMES (LinkedIn).

This lab runs on the OpenShift cluster: switch to it first with kubectx <my-openshift-context>.

Exercice 1: display Kubernetes and Openshift resources

• Retrieve etcd pod name

  Answer

  # Wait for etcd pod to be up
  kubectl  wait --timeout=240s --for=condition=Ready -n "openshift-etcd" pods -l "app=etcd,etcd=true,k8s-app=etcd"
  
  etcd_pod=$(kubectl get pods -n "openshift-etcd" -l "app=etcd,etcd=true,k8s-app=etcd" -o jsonpath='{.items[0].metadata.name}')

• Launch etcdctl --help inside etcd pod

  Answer

  # Display Kubernetes keys
  kubectl exec -t -n "openshift-etcd" "$etcd_pod" -- etcdctl --help

• Use etcdctl get ... to display Kubernetes and Openshift keys

  Answer

  # Display Kubernetes keys
  kubectl exec -t -n "openshift-etcd" "$etcd_pod" -- etcdctl get /kubernetes.io --keys-only --prefix
  
  # Display OpenShift keys (CustomResources)
  kubectl exec -t -n "openshift-etcd" "$etcd_pod" --  \
      sh -c "echo \$ETCDCTL_CERT \$ETCDCTL_KEY \$ETCDCTL_CACERT && etcdctl get /openshift.io --keys-only --prefix"

Exercice 2: perform an etcd backup and check its status

• Perform etcd snapshot with etcdctl

  Answer

  kubectl exec -t -n "openshift-etcd" "$etcd_pod" --  \
      sh -c "etcdctl \
      snapshot save /var/lib/etcd/etcd-snapshot.db"

• Check the status of the snapshot with etcdutl (etcdctl snapshot status was removed in etcd 3.6)

  Answer

  kubectl exec -t -n "openshift-etcd" "$etcd_pod" --  \
      sh -c "etcdutl \
      -w fields snapshot status /var/lib/etcd/etcd-snapshot.db"

• The backup is inside the etcd pod, find a technique to retrieve it on a secure storage.

  Answer

  # Proposal #1
  kubectl cp -n "openshift-etcd" "$etcd_pod":/var/lib/etcd/etcd-snapshot.db  ./etcd-snapshot.db
  
  # Proposal #2: check the location of '/var/lib/etcd'
  kubectl describe pod -n "openshift-etcd" "$etcd_pod"