OpenShift Security Context Constraints (SCC)

Objective

On OpenShift, the SCC admission plugin checks every pod against the Security Context Constraints the creator is allowed to use. In this lab you will:

Give a service account the rights to create pods with RBAC, and see that it is not enough
Find which SCC admitted a pod, and which SCC a pod requires
Grant an SCC to a service account
Understand why a Deployment needs the SCC on the pod service account, not on the user who creates it

Complete this lab in 30-40 minutes.

Prerequisites

An OpenShift cluster, logged in with oc as cluster-admin
The slides OPENSHIFT.pdf, section SecurityContextConstraints

Note

SCC are cluster-wide resources. On a shared cluster, each trainee works in their own namespace scc-<id> with their own service account, so the SCC grants below do not interfere.

Setup

ID=$(whoami)
NS=scc-$ID
SA=fake-user

oc new-project $NS
oc create serviceaccount $SA

# Run kubectl as the service account
alias kubectl-user="kubectl --as=system:serviceaccount:$NS:$SA -n $NS"

Create the pod manifests used in this lab:

mkdir -p $HOME/tmp

cat <<EOF > $HOME/tmp/ubuntu-simple.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-simple
  labels:
    app: ubuntu
spec:
  containers:
    - name:  ubuntu
      image: ubuntu:22.04
      command: ["sleep", "infinity"]
EOF

cat <<EOF > $HOME/tmp/ubuntu-root.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-root
  labels:
    app: ubuntu
spec:
  containers:
    - name:  ubuntu
      image: ubuntu:22.04
      command: ["sleep", "infinity"]
      securityContext:
        runAsUser: 0
EOF

cat <<EOF > $HOME/tmp/ubuntu-privileged.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-privileged
  labels:
    app: ubuntu
spec:
  containers:
    - name:  ubuntu
      image: ubuntu:22.04
      command: ["sleep", "infinity"]
      securityContext:
        privileged: true
EOF

cat <<EOF > $HOME/tmp/nginx-privileged.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-privileged
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.3
        ports:
        - containerPort: 80
        securityContext:
          privileged: true
EOF

Tip

These manifests are also generated by ex3-prereq.sh.

Exercise 1: RBAC first

Allow the service account fake-user to create pods and deployments in $NS only.

Solution

kubectl create rolebinding edit --clusterrole=edit --serviceaccount=$NS:$SA -n $NS
kubectl-user auth can-i create pods
kubectl-user auth can-i create pods -n default

Exercise 2: a pod without security requirements

Create the pod ubuntu-simple as fake-user.

Solution

kubectl-user apply -f $HOME/tmp/ubuntu-simple.yaml
kubectl-user get pods

Questions

Can fake-user use the SCC restricted-v2? And anyuid?
Which SCC admitted the pod ubuntu-simple?
Which user ID does the container run as? Where does it come from?

Solution

kubectl-user auth can-i use scc/restricted-v2
kubectl-user auth can-i use scc/anyuid

kubectl get pod ubuntu-simple -o jsonpath='{.metadata.annotations.openshift\.io/scc}{"\n"}'
kubectl get pod ubuntu-simple -o jsonpath='{.spec.containers[0].securityContext}' | jq
kubectl get pod ubuntu-simple -o jsonpath='{.spec.securityContext}' | jq
kubectl get namespace $NS -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.uid-range}{"\n"}'
kubectl wait --for=condition=Ready pod/ubuntu-simple
kubectl exec ubuntu-simple -- id
# uid=1000650000(1000650000) gid=0(root) groups=0(root),1000650000

kubectl auth can-i use scc/... prints the warning resource 'securitycontextconstraints' is not namespace scoped: it is harmless, SCCs are cluster-wide.

Every authenticated user can use restricted-v2, it is the default SCC. anyuid is not granted.
The annotation openshift.io/scc shows restricted-v2.
restricted-v2 uses the strategy MustRunAsRange: the SCC plugin injects runAsUser with the first UID of the range in the namespace annotation openshift.io/sa.scc.uid-range, for example 1000650000/10000. Each namespace gets its own range. The plugin also mutates the pod:
- container: runAsNonRoot: true, allowPrivilegeEscalation: false, all capabilities dropped
- pod: fsGroup set to the same UID, an SELinux MCS level unique to the namespace (s0:c26,c0), and the RuntimeDefault seccomp profile

Exercise 3: a pod running as root

Try to create the pod ubuntu-root as fake-user. Then:

Find which SCC this pod requires
Check whether fake-user can use it
Grant it to fake-user, check it, and create the pod

Tip

Useful commands:

oc adm policy scc-subject-review -f <podfile.yaml>: which SCC would admit this pod
oc adm policy scc-review -z <sa> -f <podfile.yaml>: which SCC would admit this pod for this service account
oc adm policy add-scc-to-user <scc-name> -z <sa>: grant an SCC to a service account

Solution

kubectl-user apply -f $HOME/tmp/ubuntu-root.yaml
# Error from server (Forbidden): ... pods "ubuntu-root" is forbidden: unable to validate against any security context constraint:
# [provider "anyuid": Forbidden: not usable by user or serviceaccount,
#  provider restricted-v2: .containers[0].runAsUser: Invalid value: 0: must be in the ranges: [1000650000, 1000659999], ...]

# As cluster-admin, all SCCs are available: shows the one which would admit the pod
oc adm policy scc-subject-review -f $HOME/tmp/ubuntu-root.yaml
# RESOURCE          ALLOWED BY
# Pod/ubuntu-root   anyuid

kubectl-user auth can-i use scc/anyuid
# no

oc adm policy add-scc-to-user anyuid -z $SA -n $NS
kubectl-user auth can-i use scc/anyuid
oc adm policy scc-review -z $SA -f $HOME/tmp/ubuntu-root.yaml -n $NS

kubectl-user apply -f $HOME/tmp/ubuntu-root.yaml
kubectl get pod ubuntu-root -o jsonpath='{.metadata.annotations.openshift\.io/scc}{"\n"}'
kubectl wait --for=condition=Ready pod/ubuntu-root
kubectl exec ubuntu-root -- id
# uid=0(root) gid=0(root) groups=0(root)

The error message lists every SCC and why it rejected the pod: it is the first place to look when a pod is refused. Just after a grant, scc-review can return an empty ALLOWED BY for a few seconds, until the RBAC cache is refreshed: run it again.

oc adm policy add-scc-to-user is a shortcut for RBAC: SCCs are granted with the use verb on the securitycontextconstraints resource. The equivalent RoleBinding, limited to the namespace, is:

oc create rolebinding scc-anyuid --clusterrole=system:openshift:scc:anyuid --serviceaccount=$NS:$SA -n $NS

Exercise 4: a privileged pod

Do the same with the pod ubuntu-privileged.

Solution

kubectl-user apply -f $HOME/tmp/ubuntu-privileged.yaml
# Error from server (Forbidden): ... [provider anyuid: .containers[0].privileged: Invalid value: true: Privileged containers are not allowed, ...]

oc adm policy scc-subject-review -f $HOME/tmp/ubuntu-privileged.yaml
oc adm policy add-scc-to-user privileged -z $SA -n $NS
oc adm policy scc-review -z $SA -f $HOME/tmp/ubuntu-privileged.yaml -n $NS

kubectl-user apply -f $HOME/tmp/ubuntu-privileged.yaml
kubectl get pod ubuntu-privileged -o jsonpath='{.metadata.annotations.openshift\.io/scc}{"\n"}'

scc-subject-review returns the first SCC which admits the pod, ordered by priority then by restrictiveness: here privileged. Grant the SCC it returns.

Exercise 5: a privileged Deployment

Create the Deployment nginx-privileged as fake-user. Are the pods created? Why?

Solution

kubectl-user apply -f $HOME/tmp/nginx-privileged.yaml
kubectl-user get deployment,replicaset,pods
kubectl-user get events --field-selector=reason=FailedCreate
# Error creating: pods "nginx-privileged-..." is forbidden: unable to validate against any security context constraint ...

The Deployment is created, but not its pods. The pods are created by the ReplicaSet controller, not by fake-user. The SCC plugin checks the SCCs available to the service account of the pod, here default, which can only use restricted-v2. See the Workflow slide.

Grant the SCC to the service account default and wait for the ReplicaSet controller to retry:

oc adm policy add-scc-to-user privileged -z default -n $NS
kubectl wait --timeout=60s --for=condition=Ready pods -l app=nginx -n $NS
kubectl-user get pods

Best practice: never grant an SCC to the default service account. Create a dedicated service account, set it in spec.template.spec.serviceAccountName, and grant the SCC to it only.

Going further

# List the SCCs, with their priority and main settings
oc get scc
# What restricted-v2 allows
oc describe scc restricted-v2

Recent OpenShift versions (seen on 4.21) add the SCC restricted-v3, which requires hostUsers: false in the pod spec: the pod runs in a Linux user namespace, so root in the container is not root on the node. It appears in the error messages above.

The safest way to run on OpenShift is to make images work under restricted-v2, instead of granting SCCs: see the lab Helm on OpenShift: Migrating to Security Context Constraints.

Clean up

oc adm policy remove-scc-from-user anyuid -z $SA -n $NS
oc adm policy remove-scc-from-user privileged -z $SA -n $NS
oc adm policy remove-scc-from-user privileged -z default -n $NS
oc delete project $NS

Reference / full solution

ex3-scc.sh: run ./ex3-prereq.sh then ./ex3-scc.sh -s.