Audit Logs & API Server
Author: Fabrice JAMMES (LinkedIn).
Objective
- Write an audit policy, which chooses what the API server records
- Enable the audit log in the API server, a static pod
- Find in the log who did what
The CKS curriculum lists “Perform deep analytical investigation and identification of malicious activity” and “Use Audit Logs to monitor access” in the Monitoring, logging and runtime security domain.
Each request to the API server is recorded at the level of the first rule of the policy which matches it:
| Level | Recorded |
|---|---|
None | Nothing |
Metadata | Who, when, which verb on which resource: no body |
Request | Metadata and the body of the request |
RequestResponse | Metadata, and the bodies of the request and of the response |
Prerequisites
The API server restarts during the lab: use a kind cluster of your own, never a shared one.
$CP is the control plane node: kind runs it as a docker container, open a shell in it with docker exec -it $CP bash. The API server is a static pod: its manifest is /etc/kubernetes/manifests/kube-apiserver.yaml on this node (see the static pods lab).
Exercise 1: the audit policy
On the control plane node, write the policy /etc/kubernetes/audit/policy.yaml, which:
- does not record the
RequestReceivedstage - records the requests on
secretsat theMetadatalevel - records the requests on
podsat theRequestResponselevel - records nothing else
Exercise 2: enable the audit log
Configure the API server, in its manifest on the control plane node, to:
- use the policy, with
--audit-policy-file - write the log in
/var/log/kubernetes/audit/audit.log, with--audit-log-path - keep the log files 7 days, at most 2 old files of at most 10 MB:
--audit-log-maxage,--audit-log-maxbackup,--audit-log-maxsize - mount the directories of the policy (read-only) and of the log, with
hostPathvolumes
Warning
Very important for the CKS exam: save a copy of the manifest outside /etc/kubernetes/manifests, e.g. in /root. The kubelet runs every file of this directory as a static pod: a backup copy there would start a second API server.
Exercise 3: investigate
Create a Secret db-password with the value s3cr3t, read it, delete it, and run a pod web (image nginx:1.27). Then find in the audit log:
- who deleted the Secret, and when
- whether the value of the Secret is in the log
- the image of the pod
web, from the log only
Tip
The log has one JSON event per line: filter it with jq 'select(...)' on .verb, .objectRef.resource, .objectRef.name, .user.username.
Bonus: who runs kubectl exec?
Run kubectl exec web -- whoami. Is it in the audit log? Change the policy so that the kubectl exec commands are recorded at the Metadata level, then find in the log who ran which command in which pod.
Tip
kubectl exec calls the subresource pods/exec: the rule on pods does not match it, and neither does it match pods/log, pods/status or pods/binding. The API server reads the policy at startup only.
Bonus: stream the audit log to a log backend
The audit log of this lab stays on the control plane node. How do you send it to a log backend (Loki, Elasticsearch, Splunk…)? No exercise here, read the two options.
Questions
- Why is
RequestResponseonsecretsa security problem? - An attacker is root on the control plane node. What can they do with the audit log?
Clean up
The apiserver-auditlogs.sh script of the k8s-advanced repository runs a similar scenario.