Horizontal Pod Autoscaling (HPA)

Objectives

Use the Horizontal Pod Autoscaler (HPA) to automatically scale the number of pod replicas based on CPU usage, so the workload matches demand instead of paying for idle capacity.

Prerequisites

Understanding HPA

The Horizontal Pod Autoscaler scales a workload out (more replicas) or in (fewer replicas) by watching a metric such as CPU utilization. It is one of the core FinOps levers on Kubernetes: capacity follows demand.

Q1: What does the HPA need to work?

Answer
  • A metrics source (metrics-server) to read pod CPU/memory usage.
  • Resource requests defined on the container — utilization is computed as usage / request.
  • A scalable target (Deployment, ReplicaSet, StatefulSet).

The HPA controller periodically computes:

desiredReplicas = ceil(currentReplicas × currentMetricValue / targetMetricValue)

Q2: When should you use HPA vs VPA?

Answer
  • HPA is best for stateless services with variable traffic: more load → more replicas.
  • VPA is best for right-sizing requests of workloads with a stable replica count.
  • Avoid running both on the same CPU/memory metric — they fight each other.

Setup

The cluster is shared by the trainees: metrics-server are installed once for the whole cluster by the trainer, with install.sh. Do not run it yourself.

Info

Trainer only, once per cluster, before the lab:

git clone https://github.com/k8s-school/k8s-advanced
cd k8s-advanced/labs/8_finops
./install.sh

The script targets the kind-kind cluster, and creates it if it does not exist (CLUSTER_NAME=<name> for another one). It can be run again safely: a running VPA is never reinstalled.

Switch to the shared kind cluster:

kubectx kind-kind

Verify metrics are available:

kubectl top nodes
kubectl top pods -A

Deploy a CPU-bound workload

We use the classic php-apache image, which burns CPU on every HTTP request.

Shared cluster: run the lab in a namespace suffixed by your user name so several students can work on the same cluster without collisions.

export NS="hpa-demo-$USER"
kubectl create namespace "$NS"
kubectl config set-context --current --namespace="$NS"

kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
kubectl rollout status deployment/php-apache

Note: The Deployment defines cpu: 200m as a request and cpu: 500m as a limit. The request is what the HPA uses as the 100% reference for utilization.

Create the HorizontalPodAutoscaler

Q3: Create an HPA targeting 50% average CPU, between 1 and 3 replicas.

Solution
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=3

# Inspect it
kubectl get hpa php-apache

The first reading may show <unknown> for a few seconds until metrics-server reports usage:

# Wait until TARGETS shows a percentage instead of <unknown>
kubectl get hpa php-apache -w

Generate load and watch it scale OUT

Open two terminals (or use byobu).

Terminal 1 — watch the autoscaler and the deployment:

kubectl get hpa php-apache -w

Terminal 2 — generate load:

kubectl run load-generator --image=busybox:1.36 --restart=Never -- \
  /bin/sh -c "while true; do wget -q -O- http://php-apache; done"

After 1–2 minutes the CPU target is exceeded and the HPA increases the replica count:

kubectl get deployment php-apache
kubectl get pods -l run=php-apache

Expected: TARGETS climbs well above 50% and REPLICAS grows to the maximum. The maximum is kept low (3, each pod can use up to 500m CPU) because all the trainees share the same cluster.

Stop the load and watch it scale IN

kubectl delete pod load-generator

After the stabilization window (default ~5 minutes for scale-down) the replica count drops back to 1:

kubectl get hpa php-apache -w

FinOps takeaway: the scale-down stabilization window prevents flapping, but it also means you keep paying for extra replicas a few minutes after the spike ends. Tune behavior.scaleDown.stabilizationWindowSeconds to balance cost vs responsiveness.

Automated run

The solution of this lab is validated by ex1-hpa.sh.

Cleanup

kubectl delete namespace "$NS"
kubectl config set-context --current --namespace=default

Troubleshooting

# TARGETS stuck at <unknown> ?  metrics-server is not serving metrics:
kubectl top pods -n "$NS"
kubectl logs -n kube-system deployment/metrics-server

# On kind, metrics-server must run with --kubelet-insecure-tls (handled by install.sh)
kubectl get deployment metrics-server -n kube-system -o yaml | grep -A5 args