Pod Security Admission

Author: Fabrice JAMMES (LinkedIn).

Objective

Apply the Pod Security Standards to a namespace with the Pod Security Admission controller:

LevelMeaning
privilegedNo restriction
baselineBlocks the known privilege escalations: privileged, host namespaces, hostPath, extra capabilities…
restrictedHardened pods: non-root user, allowPrivilegeEscalation: false, all capabilities dropped, seccomp profile

Each level is applied in one or more modes, with a namespace label pod-security.kubernetes.io/<mode>=<level>:

ModeEffect on a pod which violates the level
enforceRejected
auditAccepted, with an annotation in the audit log
warnAccepted, with a warning for the user

The lab works on a cluster shared by the trainees: the levels are set on a namespace of your own. Do the security context lab first.

Prerequisites

# The shared kind cluster (on the CKS training server: kind export kubeconfig, then kubectx kind-kind)
kubectx kind-kind
NS=verify-pod-security-$(whoami)
kubectl create namespace $NS
kubectl config set-context --current --namespace=$NS

Write this privileged pod in $HOME/tmp/ubuntu-privileged.yaml:

mkdir -p $HOME/tmp
cat <<EOF > $HOME/tmp/ubuntu-privileged.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-privileged
  labels:
    app: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu:22.04
    command: ["sleep", "infinity"]
    securityContext:
      privileged: true
EOF

Exercise 1 (optional): is Pod Security Admission enabled?

Check that the PodSecurity admission plugin is enabled, with kube-apiserver -h.

Solution
APISERVER_POD=$(kubectl -n kube-system get pods -l component=kube-apiserver -o jsonpath='{.items[0].metadata.name}')
kubectl -n kube-system exec $APISERVER_POD -- kube-apiserver -h | grep "default enabled plugins" | grep -o PodSecurity

PodSecurity is enabled by default since Kubernetes 1.23 (GA in 1.25), it replaced the PodSecurityPolicies.

Exercise 2: enforce the restricted level

Enforce the restricted level on your namespace, and audit on restricted. Then try to deploy the privileged pod.

Solution
kubectl label --overwrite namespace $NS \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted
kubectl apply -f $HOME/tmp/ubuntu-privileged.yaml || echo "EXPECTED ERROR"
# Error from server (Forbidden): error when creating ".../ubuntu-privileged.yaml": pods "ubuntu-privileged" is forbidden:
# violates PodSecurity "restricted:latest": privileged (container "ubuntu" must not set securityContext.privileged=true),
# allowPrivilegeEscalation != false (...), unrestricted capabilities (...), runAsNonRoot != true (...), seccompProfile (...)

The error lists all the fields to change. kubectl apply --dry-run=server runs the admission controllers without creating the pod: a good way to test a manifest against a namespace.

Exercise 3: privileged level, with warnings

Enforce the privileged level, with warnings and audit on the baseline level. Deploy the privileged pod again.

Solution
kubectl label --overwrite namespace $NS \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/warn=baseline \
  pod-security.kubernetes.io/audit=baseline
kubectl apply -f $HOME/tmp/ubuntu-privileged.yaml
# Warning: would violate PodSecurity "baseline:latest": privileged (container "ubuntu" must not set securityContext.privileged=true)
# pod/ubuntu-privileged created

The pod is created, but kubectl displays a warning. The audit mode adds the same information as an annotation to the event of the audit log: it is only visible if the API server has an audit policy, see the audit logs lab.

warn and audit are the way to roll out a level: first see which workloads would be rejected, then enforce.

Exercise 4: before tightening the namespace

The privileged pod is running. Check which running pods would violate the restricted level, without changing the namespace.

Solution
kubectl label --dry-run=server --overwrite namespace $NS pod-security.kubernetes.io/enforce=restricted
# Warning: existing pods in namespace "verify-pod-security-..." violate the new PodSecurity enforce level "restricted:latest"
# Warning: ubuntu-privileged: privileged, allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile

The running pods are never evicted by a new label: only the new pods are checked. A Deployment whose pods violate the level keeps running, but cannot re-create its pods any more.

Exercise 5: a restricted ubuntu pod

Enforce the restricted level, with warnings and audit on restricted. Then update the ubuntu pod specification so that it runs in the namespace, with the NET_BIND_SERVICE capability enabled. Use the Pod Security Standards page.

Tip

The ubuntu image runs as root: choose a non-root uid, for example 65534 (the nobody user of the image).

Solution
kubectl delete pod ubuntu-privileged
kubectl label --overwrite namespace $NS \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted

cat <<EOF > $HOME/tmp/ubuntu-restricted.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-restricted
  labels:
    app: ubuntu
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65534
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: ubuntu
    image: ubuntu:22.04
    command: ["sleep", "infinity"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]
        add: ["NET_BIND_SERVICE"]
EOF
kubectl apply -f $HOME/tmp/ubuntu-restricted.yaml
kubectl wait --for=condition=Ready pod/ubuntu-restricted
kubectl exec ubuntu-restricted -- id
# uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup)

restricted requires: no privileged container, allowPrivilegeEscalation: false, a non-root user, a RuntimeDefault or Localhost seccomp profile, and drop: ["ALL"]. NET_BIND_SERVICE is the only capability it allows to add back.

Exercise 6: play with the warn label

Enforce baseline and warn on restricted. Then create a copy of the restricted pod, named ubuntu-chown, which also adds the CHOWN capability. Is it created? What does the warning say? And with enforce=restricted?

Solution
kubectl label --overwrite namespace $NS \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/warn=restricted
sed -e 's/ubuntu-restricted/ubuntu-chown/' -e 's/add: \["NET_BIND_SERVICE"\]/add: ["NET_BIND_SERVICE", "CHOWN"]/' \
  $HOME/tmp/ubuntu-restricted.yaml > $HOME/tmp/ubuntu-chown.yaml
kubectl apply -f $HOME/tmp/ubuntu-chown.yaml
# Warning: would violate PodSecurity "restricted:latest": unrestricted capabilities (container "ubuntu" must not include "CHOWN" in securityContext.capabilities.add)
# pod/ubuntu-chown created

kubectl delete pod ubuntu-chown
kubectl label --overwrite namespace $NS pod-security.kubernetes.io/enforce=restricted
kubectl apply -f $HOME/tmp/ubuntu-chown.yaml || echo "EXPECTED ERROR"
# Error from server (Forbidden): ... violates PodSecurity "restricted:latest": unrestricted capabilities ...

CHOWN is in the default capabilities of the container runtimes: baseline accepts it, restricted does not. The same message is a warning with warn, an error with enforce.

Questions

  1. Who can bypass the Pod Security Admission of a namespace?
  2. How does OpenShift do the same?
Answers
  1. Anybody allowed to change the labels of the namespace: protect the update and patch verbs on namespaces with RBAC. The API server can also exempt users, runtime classes or namespaces (AdmissionConfiguration), and set a cluster-wide default level.
  2. OpenShift uses the Security Context Constraints (SCC): they validate the pods and mutate them (random uid of the namespace range, for example). OpenShift also syncs the Pod Security Admission labels of the namespaces from the SCCs of their service accounts. See the SCC lab.

Clean up

kubectl delete namespace $NS
kubectl config set-context --current --namespace=default

The solution of this lab is validated by ex2-podsecurity.sh.