Hardening with CIS Benchmarks - Security Remediations

Author: Fabrice JAMMES (LinkedIn).

Objective

Fix failed checks of the kube-bench report by editing the static pod manifests of the control plane, then check the fix with a new scan. It is the typical CKS task: “fix the issues reported by kube-bench on the control plane”.

Prerequisites

The control plane components restart during the lab: use a kind cluster of your own, never a shared one.

# On the CKS training server, use your own cluster ($CLUSTER_NAME),
# elsewhere create a cluster for the lab
CLUSTER=${CLUSTER_NAME:-kube-bench-$(whoami)}
kind get clusters | grep -qx $CLUSTER || kind create cluster --name $CLUSTER
kind export kubeconfig --name $CLUSTER
CP=$CLUSTER-control-plane
kubectl create namespace kube-bench
kubectl config set-context --current --namespace=kube-bench
mkdir -p $HOME/kube-bench && cd $HOME/kube-bench

The Job of the previous lab, which scans the control plane, and a function which runs it again and displays the result of some checks:

cat <<EOF > kube-bench-master.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-master
spec:
  template:
    spec:
      hostPID: true
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule
      containers:
      - name: kube-bench
        image: docker.io/aquasec/kube-bench:v0.14.1
        command: ["kube-bench", "run", "--targets", "master"]
        volumeMounts:
        - {name: var-lib-etcd, mountPath: /var/lib/etcd, readOnly: true}
        - {name: var-lib-kubelet, mountPath: /var/lib/kubelet, readOnly: true}
        - {name: etc-systemd, mountPath: /etc/systemd, readOnly: true}
        - {name: lib-systemd, mountPath: /lib/systemd/, readOnly: true}
        - {name: etc-kubernetes, mountPath: /etc/kubernetes, readOnly: true}
        - {name: usr-bin, mountPath: /usr/local/mount-from-host/bin, readOnly: true}
      restartPolicy: Never
      volumes:
      - {name: var-lib-etcd, hostPath: {path: /var/lib/etcd}}
      - {name: var-lib-kubelet, hostPath: {path: /var/lib/kubelet}}
      - {name: etc-systemd, hostPath: {path: /etc/systemd}}
      - {name: lib-systemd, hostPath: {path: /lib/systemd}}
      - {name: etc-kubernetes, hostPath: {path: /etc/kubernetes}}
      - {name: usr-bin, hostPath: {path: /usr/bin}}
EOF

# Usage: scan <check id>...
scan() {
  kubectl delete job kube-bench-master --ignore-not-found >/dev/null
  kubectl apply -f kube-bench-master.yaml >/dev/null
  kubectl wait --for=condition=complete job/kube-bench-master --timeout=300s >/dev/null
  for check in "$@"; do
    kubectl logs job/kube-bench-master | grep "^\[[A-Z]*\] $check "
  done
}
scan 1.4.1 1.3.2
# [FAIL] 1.4.1 Ensure that the --profiling argument is set to false (Automated)
# [FAIL] 1.3.2 Ensure that the --profiling argument is set to false (Automated)

Exercise 1: the profiling of the scheduler

Check 1.4.1: the scheduler serves profiling data (/debug/pprof), which shows details on the internals of the cluster and can be used to overload it.

  1. Which is the default value of --profiling for the scheduler? Is the flag set in its manifest?
  2. Set --profiling=false in the manifest of the scheduler, on the control plane node. Save a copy of the manifest first, outside /etc/kubernetes/manifests.
  3. Scan again.
Warning

Very important for the CKS exam: save a copy of the manifest outside /etc/kubernetes/manifests, e.g. in /root. The kubelet runs every file of this directory as a static pod: a backup copy there would start a second scheduler.

Solution
kubectl exec -n kube-system kube-scheduler-$CP -- kube-scheduler --help | grep -A1 -- "--profiling"
#       --profiling                 DEPRECATED: enable profiling via web interface host:port/debug/pprof/ ... (default true)
# In the exam, read the command line with: kubectl describe pod -n kube-system kube-scheduler-$CP
kubectl get pod -n kube-system kube-scheduler-$CP -o jsonpath='{.spec.containers[0].command}' | tr ',' '\n' | grep -- --profiling || echo "not set"
# not set

In the exam, edit the manifest with vi, on the control plane node:

# In the exam: ssh to the control plane node instead
docker exec -it $CP bash
# In the node
cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.orig
apt-get update && apt-get install -y vim
vi /etc/kubernetes/manifests/kube-scheduler.yaml
# Add the line "    - --profiling=false" after "    - kube-scheduler"

Or with sed, from the server:

# In the exam: ssh to the control plane node, then run the command without docker exec
docker exec $CP cp /etc/kubernetes/manifests/kube-scheduler.yaml /root/kube-scheduler.yaml.orig
docker exec $CP sed -i 's|^    - kube-scheduler$|&\n    - --profiling=false|' /etc/kubernetes/manifests/kube-scheduler.yaml
# Check the change against the backup
# diff exits with code 1 when the files differ, which is the expected case here:
# "|| true" prevents a script running with "set -e" (like the CI of this page) from stopping
docker exec $CP diff -u /root/kube-scheduler.yaml.orig /etc/kubernetes/manifests/kube-scheduler.yaml || true

The diff shows the new flag:

@@ -12,2 +12,3 @@
     - kube-scheduler
+    - --profiling=false
     - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf

The kubelet re-creates the scheduler pod: wait until the new pod runs with the flag, then scan again.

until kubectl get pod -n kube-system kube-scheduler-$CP -o jsonpath='{.spec.containers[0].command}' 2>/dev/null | grep -q profiling=false; do sleep 5; done
kubectl wait -n kube-system --for=condition=Ready pod/kube-scheduler-$CP --timeout=120s
scan 1.4.1
# [PASS] 1.4.1 Ensure that the --profiling argument is set to false (Automated)

If the scheduler does not come back, the YAML is not valid. Read its logs, or restore the copy:

# In the exam: ssh to the control plane node, then run the commands without docker exec
docker exec $CP sh -c 'crictl ps -a | grep scheduler'
docker exec $CP crictl logs <container-id>
docker exec $CP cp /root/kube-scheduler.yaml.orig /etc/kubernetes/manifests/kube-scheduler.yaml

Exercise 2: the controller manager

Fix the check 1.3.2 alone: the same flag, for the controller manager.

Solution
# In the exam: ssh to the control plane node, then run the command without docker exec
docker exec $CP cp /etc/kubernetes/manifests/kube-controller-manager.yaml /root/kube-controller-manager.yaml.orig
docker exec $CP sed -i 's|^    - kube-controller-manager$|&\n    - --profiling=false|' /etc/kubernetes/manifests/kube-controller-manager.yaml
# Check the change against the backup (|| true: diff exits with code 1 when the files differ)
docker exec $CP diff -u /root/kube-controller-manager.yaml.orig /etc/kubernetes/manifests/kube-controller-manager.yaml || true
until kubectl get pod -n kube-system kube-controller-manager-$CP -o jsonpath='{.spec.containers[0].command}' 2>/dev/null | grep -q profiling=false; do sleep 5; done
kubectl wait -n kube-system --for=condition=Ready pod/kube-controller-manager-$CP --timeout=120s
scan 1.3.2
# [PASS] 1.3.2 Ensure that the --profiling argument is set to false (Automated)

The same method fixes 1.2.15 for the API server. For the audit log checks (1.2.16 to 1.2.19), see the audit logs lab, and for the encryption of the Secrets (1.2.27, 1.2.28), the etcd encryption lab.

Questions

  1. Why save the copy of a manifest outside /etc/kubernetes/manifests?
  2. The fix of a kubelet check (target node) is in which file?
Answers
  1. The kubelet runs every file of this directory as a static pod: a copy there would start a second scheduler, or fail on a duplicate name. See the static pods lab.
  2. In the configuration file of the kubelet, /var/lib/kubelet/config.yaml on kubeadm clusters (its path is given by --config in the command line of the kubelet), or in its flags (/etc/systemd/system/kubelet.service.d/). The kubelet is not a static pod: restart it with systemctl restart kubelet.

Clean up

# In the exam: ssh to the control plane node, then run the command without docker exec
docker exec $CP cp /root/kube-scheduler.yaml.orig /etc/kubernetes/manifests/kube-scheduler.yaml
docker exec $CP cp /root/kube-controller-manager.yaml.orig /etc/kubernetes/manifests/kube-controller-manager.yaml
kubectl delete namespace kube-bench
kubectl config set-context --current --namespace=default
cd $HOME && rm -rf $HOME/kube-bench
# Delete the cluster only if it was created for the lab
if [ -z "${CLUSTER_NAME:-}" ]; then kind delete cluster --name $CLUSTER; fi

Next: Hardening with CIS Benchmarks - Automation and Continuous Compliance.