Monitoring with Prometheus

Author: Fabrice JAMMES (LinkedIn).

Kubernetes Monitoring Lab with Helm

This lab runs on the kind cluster shared by all trainees: switch to it first with kubectx kind-kind.

Objective

The goal of this lab is to deploy the Prometheus Stack using Helm while following best practices in shell scripting.

Prerequisites

Ensure you have the following installed:

  • Kubernetes cluster (e.g., Minikube, Kind, or a cloud-managed cluster)
  • kubectl configured to access the cluster
  • helm installed
  • bash shell

Steps to Deploy Prometheus Stack

Note

The Prometheus installation is completed before the lab; the following instructions are for reference only.

Install Prometheus Stack

git clone https://github.com/k8s-school/demo-prometheus
cd demo-prometheus
./install.sh

6. Access Prometheus and Grafana

Once the deployment is complete, follow these exercises to interact with the monitoring stack:

  1. Watch all pods in the monitoring namespace:
Answer
kubectl get pods -n monitoring --watch
  1. Find the Prometheus operator and the CRDs it manages:

Which Custom Resource Definitions were installed by the stack? Which pod is the operator, and which objects does it reconcile into StatefulSets?

Answer
# CRDs installed by the Prometheus operator (API group monitoring.coreos.com)
kubectl get crd | grep monitoring.coreos.com
kubectl api-resources --api-group=monitoring.coreos.com

# The operator itself is a plain Deployment
kubectl get deployments -n monitoring | grep operator

# Custom resources watched by the operator
kubectl get prometheuses,alertmanagers,servicemonitors,prometheusrules -n monitoring

# The Prometheus StatefulSet is created by the operator, from the Prometheus custom resource
kubectl get statefulsets -n monitoring
kubectl get statefulset -n monitoring prometheus-prometheus-stack-kube-prom-prometheus \
  -o jsonpath='{.metadata.ownerReferences[*].kind}{"/"}{.metadata.ownerReferences[*].name}{"\n"}'
  1. Retrieve Grafana password:
Answer
# Add helm repository
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts || echo "Unable to add repo prometheus-community"
helm repo add stable https://charts.helm.sh/stable --force-update
helm repo update

# Query helm configuration, for the chart version installed by demo-prometheus
helm show values --version 69.3.0 prometheus-community/kube-prometheus-stack | grep adminPassword

# Or read it from the Grafana secret
kubectl get secret -n monitoring prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d
  1. Port forward Prometheus web UI to access it in a web browser:
Answer
kubectl port-forward -n monitoring prometheus-prometheus-stack-kube-prom-prometheus-0 9090

Eventually create a ssh tunnel if Kubernetes is secured behind a ssh bastion. On a server shared by several trainees, pick a local port of your own (e.g. 909<ID>:9090).

Then open your browser and go to: http://localhost:9090

Check the metrics and the alert rules.

  1. Port forward Grafana to access it in a web browser:

Retrieve the Grafana port using kubectl get svc ....

Answer
# port-forward to svc
kubectl port-forward svc/prometheus-stack-grafana -n monitoring 8080:80

# or port-forward to pod
kubectl port-forward $(kubectl get  pods --selector=app.kubernetes.io/name=grafana -n  monitoring --output=jsonpath="{.items..metadata.name}") -n monitoring  3000

Eventually create a ssh tunnel if Kubernetes is secured behing a ssh bastion.

Then open your browser and go to: http://localhost:8080

Check the Kubernetes dashboards.

Conclusion

This lab guides you through deploying a monitoring stack using Helm and Kubernetes, allowing you to explore Prometheus and Grafana for cluster monitoring.

The prometheus demo is available here: https://github.com/k8s-school/demo-prometheus