Seccomp Security Profiles

Author: Fabrice JAMMES (LinkedIn).

Objective

Restrict the system calls of a container with seccomp (secure computing mode): a filter in the Linux kernel which allows, denies or logs each system call of a process.

Profile typeWhere is the profile?Use
UnconfinedNone: no filterThe default of the kubelet, unless --seccomp-default is set
RuntimeDefaultBuilt in the container runtime (containerd, CRI-O)A good default: denies about 50 dangerous system calls
LocalhostA JSON file on the node, under /var/lib/kubelet/seccomp/A profile for one application

The CKS curriculum lists “Use kernel hardening tools such as AppArmor, seccomp” in the System hardening domain.

Prerequisites

The lab writes files on the nodes: use a kind cluster of your own, never a shared one.

# On the CKS training server, use your own cluster ($CLUSTER_NAME),
# elsewhere create a cluster for the lab
CLUSTER=${CLUSTER_NAME:-seccomp-$(whoami)}
kind get clusters | grep -qx $CLUSTER || kind create cluster --name $CLUSTER
kind export kubeconfig --name $CLUSTER
kubectl create namespace seccomp
kubectl config set-context --current --namespace=seccomp

Exercise 1: the default profile

Run two pods with the image busybox:1.36 and the command sleep infinity: default, without any security context, and runtime-default, with the RuntimeDefault seccomp profile. In each pod, read the Seccomp field of /proc/1/status, then run unshare -U true, which creates a user namespace.

Solution
kubectl run default --image=busybox:1.36 -- sleep infinity
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: runtime-default
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: main
    image: busybox:1.36
    command: ["sleep", "infinity"]
EOF
kubectl wait --for=condition=Ready pod/default pod/runtime-default --timeout=120s

kubectl exec default -- grep Seccomp: /proc/1/status
# Seccomp:	0
kubectl exec runtime-default -- grep Seccomp: /proc/1/status
# Seccomp:	2
kubectl exec default -- unshare -U true && echo "allowed"
# allowed
kubectl exec runtime-default -- unshare -U true || echo "EXPECTED ERROR"
# unshare: unshare(0x10000000): Operation not permitted

Seccomp: 0 means no filter, 2 means a filter is active. Without a security context, the container runs Unconfined: it can call unshare, used by many container escapes. The RuntimeDefault profile denies it.

Set the profile at the pod level (spec.securityContext), it applies to all its containers; a container can override it in its own securityContext. The restricted level of the Pod Security Standards requires RuntimeDefault or Localhost.

Exercise 2: a custom profile

Write the profile no-mkdir.json, which allows all the system calls except mkdir and mkdirat, which return an error. Copy it on all the nodes, in /var/lib/kubelet/seccomp/profiles/. Then run a pod no-mkdir which uses it, and try to create a directory, then a file, in /tmp.

Tip

A profile is a JSON document with a defaultAction and a list of syscalls with their action: SCMP_ACT_ALLOW, SCMP_ACT_ERRNO, SCMP_ACT_LOG… The path in the pod is relative to /var/lib/kubelet/seccomp/.

Solution
for node in $(kind get nodes --name $CLUSTER); do
  # In the exam: ssh to the node, then run the command without docker exec
  docker exec $node mkdir -p /var/lib/kubelet/seccomp/profiles
  cat <<EOF | docker exec -i $node tee /var/lib/kubelet/seccomp/profiles/no-mkdir.json
{
  "defaultAction": "SCMP_ACT_ALLOW",
  "syscalls": [
    {
      "names": ["mkdir", "mkdirat"],
      "action": "SCMP_ACT_ERRNO"
    }
  ]
}
EOF
done

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: no-mkdir
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/no-mkdir.json
  containers:
  - name: main
    image: busybox:1.36
    command: ["sleep", "infinity"]
EOF
kubectl wait --for=condition=Ready pod/no-mkdir --timeout=120s
kubectl exec no-mkdir -- mkdir /tmp/new-dir || echo "EXPECTED ERROR"
# mkdir: can't create directory '/tmp/new-dir': Operation not permitted
kubectl exec no-mkdir -- touch /tmp/new-file && echo "allowed"
# allowed

The kubelet reads the profile on the node where the pod runs: a missing file on one node makes the pods of this node fail, hence the copy on all the nodes. In the exam, the task often gives a profile to copy in /var/lib/kubelet/seccomp/ of a node, with scp or ssh.

A real profile does the opposite: defaultAction: SCMP_ACT_ERRNO and the list of the system calls the application needs. Writing it by hand is long: the Security Profiles Operator records the system calls of an application and generates the profile.

Exercise 3: a missing profile

Run a pod missing-profile whose Localhost profile is profiles/missing.json, a file which does not exist. What is the status of the pod? Where is the error?

Solution
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: missing-profile
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/missing.json
  containers:
  - name: main
    image: busybox:1.36
    command: ["sleep", "infinity"]
EOF
sleep 15
kubectl get pod missing-profile
# NAME              READY   STATUS                 RESTARTS   AGE
# missing-profile   0/1     CreateContainerError   0          15s
kubectl get events --field-selector involvedObject.name=missing-profile,reason=Failed -o custom-columns=MESSAGE:.message
# Error: failed to create containerd container: cannot load seccomp profile
#   "/var/lib/kubelet/seccomp/profiles/missing.json": open ...: no such file or directory

The API server accepts the pod: the profile is only read by the container runtime, on the node. The error is in the events of the pod.

Questions

  1. How do you apply RuntimeDefault to all the pods of a cluster, without changing their manifests?
  2. What is the difference with AppArmor?
Answers
  1. Start the kubelets with --seccomp-default (or seccompDefault: true in their configuration file): the pods without a profile get RuntimeDefault instead of Unconfined. Test the applications first: a few need a system call that RuntimeDefault denies.
  2. Seccomp filters system calls, whatever their arguments: “this process cannot call mkdir”. AppArmor filters access to resources, by path: “this process cannot write in /etc”. Both are often used together.

Clean up

kubectl delete namespace seccomp
kubectl config set-context --current --namespace=default
for node in $(kind get nodes --name $CLUSTER); do
  # In the exam: ssh to the node, then run the command without docker exec
  docker exec $node rm -f /var/lib/kubelet/seccomp/profiles/no-mkdir.json
done
# Delete the cluster only if it was created for the lab
if [ -z "${CLUSTER_NAME:-}" ]; then kind delete cluster --name $CLUSTER; fi

The seccomp.sh script of the k8s-advanced repository runs more profiles, on a cluster it creates.