Encrypt Secrets at rest in etcd
Author: Fabrice JAMMES (LinkedIn).
Objective
- Read a
Secretdirectly in etcd, and see that it is only base64-encoded, not encrypted - Configure the API server to encrypt the Secrets before writing them in etcd
- Encrypt the existing Secrets
See Encrypting Confidential Data at Rest.
Prerequisites
The API server restarts during the lab: use a kind cluster of your own, never a shared one.
$CP is the control plane node: kind runs it as a docker container, and etcd runs in it as a static pod.
Exercise 1: read a Secret in etcd
Create a Secret s1 in the default namespace, with the key password and the value my-secret-password.
Then read it with etcdctl in the etcd pod. The key of a Secret in etcd is /registry/secrets/<namespace>/<name>.
Tip
etcdctl needs the client certificates of etcd: they are in /etc/kubernetes/pki/etcd/ on the control plane node, look at the etcd static pod manifest to find them.
Exercise 2: configure encryption at rest
Write the encryption configuration
On the control plane node, create the file /etc/kubernetes/enc/enc.yaml: an EncryptionConfiguration which encrypts the secrets resource with the aescbc provider and a random 32-byte key, and still reads the unencrypted data (identity provider).
Configure the API server
The API server is a static pod: edit its manifest /etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node, to:
- add the flag
--encryption-provider-config=/etc/kubernetes/enc/enc.yaml - mount the directory
/etc/kubernetes/encof the node in the pod (hostPathvolume)
Warning
Very important for the CKS exam: save a copy of the manifest outside /etc/kubernetes/manifests, e.g. in /root. The kubelet runs every file of this directory as a static pod: a backup copy there would start a second API server.
Exercise 3: check the encryption
Create a Secret s2, then read s1 and s2 in etcd. Which one is encrypted? How do you encrypt the other one?
Questions
- Where is the encryption key? Who can read it?
- Is an etcd backup of this cluster still a sensitive file?
- How do you rotate the key?
Clean up
On the CKS training server, your cluster keeps the encryption at rest: it is harmless for the next labs.