Operators with OLM: the Grafana Operator

Author: Fabrice JAMMES (LinkedIn). Duration: 15 minutes of demo, 30-40 minutes of lab

Objective

On OpenShift, operators are installed and upgraded by the Operator Lifecycle Manager (OLM), from the catalogs displayed in the web console. This page has two parts:

1. Demo, by the trainer: install the Grafana Operator from the web console, in All namespaces mode.
2. Lab, by the trainees: find the objects OLM created (Subscription, InstallPlan, ClusterServiceVersion, OperatorGroup), create Grafana instances in their own namespace, and check that the operator keeps reconciling them.

Both parts run on the OpenShift cluster shared by all trainees: switch to it first with kubectx <my-openshift-context>.

Part 1 — Demo: install the Grafana Operator from the web console

The OLM objects

In the order they appear during the demo:

The catalog publishes packages. The Subscription picks one, in a namespace that has an OperatorGroup. OLM computes an InstallPlan; once approved, it creates the CSV and the CRDs. OLM then deploys the operator described in the CSV. On an upgrade, a new version appears in the channel, OLM creates a new InstallPlan and, once approved, a new CSV replaces the old one.

Before starting

Logged in to the web console as kubeadmin, in the Administrator perspective. Since OpenShift 4.20, the Operators menu is named Ecosystem and the OperatorHub is the Software Catalog; older versions use Operators > OperatorHub.

Keep a terminal open next to the console, to show the objects each click creates (-w only accepts a single resource type, hence watch):

watch -n 2 oc get subscription,installplan,csv -n openshift-operators

1. The catalogs

Administration > Cluster Settings > Configuration > OperatorHub > Sources: each catalog is a CatalogSource, a pod serving an index of operators.

oc get catalogsources -n openshift-marketplace
oc get pods -n openshift-marketplace
oc get packagemanifests -n openshift-marketplace | wc -l

Explain the four sources: Red Hat (supported by Red Hat), Certified (tested by Red Hat, supported by the vendor), Community (no support) and Marketplace (paid).

2. Find the operator

Ecosystem > Software Catalog, filter on the Community source and search for Grafana. Open the Grafana Operator card (the console warns that community operators are not supported by Red Hat) and show:

• the capability level, from Basic Install to Auto Pilot
• the channel (v5) and the available versions. A channel is an update stream published by the operator author: the Subscription follows one channel, and OLM proposes the new versions released in it. Here v5 is the major version of the operator (5.25.0), not of Grafana itself; other operators use stable, fast or candidate. Changing major version means switching channel, on purpose.
• the source (Community) and the provider

The panel does not list the provided APIs (the CRDs): they appear on the next page, after clicking Install, and later in Installed Operators > Grafana Operator > Details. The CLI shows them too, with the channels:

oc get packagemanifest grafana-operator -n openshift-marketplace \
  -o jsonpath='{range .status.channels[*]}{.name}{" "}{.currentCSV}{"\n"}{end}'
oc get packagemanifest grafana-operator -n openshift-marketplace \
  -o jsonpath='{range .status.channels[?(@.name=="v5")].currentCSVDesc.customresourcedefinitions.owned[*]}{.kind}{"\n"}{end}'

3. Install it

Click Install and fill in the form:

• Update channel: v5
• Installation mode: All namespaces on the cluster, the operator goes to openshift-operators and watches every namespace. Point out that the operator lists the supported modes: some operators can only watch their own namespace.
• Update approval: Manual, so that every new version waits for an administrator

Click Install. The terminal shows a Subscription, then an InstallPlan with APPROVED false: nothing is installed yet.

4. Approve the InstallPlan

The console displays Manual approval required: click Review manual InstallPlan. The InstallPlan lists everything OLM is about to create: the CRDs, the ServiceAccount, the ClusterRoles and the operator Deployment. Click Approve.

The CSV goes through Pending, InstallReady, Installing, then Succeeded.

5. Tour of the installed operator

Ecosystem > Installed Operators > Grafana Operator:

• Details: the status of the CSV, the provided APIs
• YAML: the ClusterServiceVersion, which holds the operator Deployment, the owned CRDs and the RBAC it needs
• Subscription: the channel, the approval mode, the installed version; this is where an upgrade is approved, or the approval mode changed
• Events

Then Workloads > Pods, project openshift-operators: the operator is a plain pod. Finally switch to any other project: Installed Operators also lists the Grafana Operator there, because OLM copies the CSV in every namespace in All namespaces mode.

6. The same installation from the CLI

cat <<EOF | oc apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: grafana-operator
  namespace: openshift-operators
spec:
  channel: v5
  name: grafana-operator
  source: community-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Automatic
EOF
oc get subscription grafana-operator -n openshift-operators -w

With Automatic, OLM approves the InstallPlan itself, now and for every new version published in the channel. Manual keeps upgrades under control, at the cost of an approval for each version:

IP=$(oc get subscription grafana-operator -n openshift-operators -o jsonpath='{.status.installPlanRef.name}')
oc patch installplan "$IP" -n openshift-operators --type merge -p '{"spec":{"approved":true}}'

OLM resolves the Subscriptions of a namespace together: one Manual Subscription in openshift-operators can hold back the upgrades of the other operators of this namespace.

7. Uninstall, at the end of the training

Installed Operators > Grafana Operator > Actions > Uninstall Operator. It removes the Subscription and the CSV, so the operator Deployment goes away, but OLM keeps the CRDs and the custom resources of every namespace. Remove the CRDs explicitly:

oc get crd -o name | grep grafana.integreatly.org | xargs -r oc delete

The solutions are validated by ex1-olm-grafana.sh: -s installs the operator, -c uninstalls it and removes its CRDs.

Part 2 — Lab

Setup

NS="grafana-$USER"
oc new-project "$NS"

Exercise 1 — The objects created by OLM

1. Which Subscription installed the Grafana Operator, from which catalog and which channel? Is the upgrade approval automatic or manual?

oc get subscription -n openshift-operators
oc get subscription grafana-operator -n openshift-operators -o yaml

oc get catalogsources -n openshift-marketplace
oc get installplan -n openshift-operators

2. Which version of the operator is installed? Which CRDs does it own?

# The CSVs installed in the operator namespace, with their version and phase
oc get csv -n openshift-operators
# The CSV currently installed by the Subscription
CSV=$(oc get subscription grafana-operator -n openshift-operators -o jsonpath='{.status.installedCSV}')
# The CRDs this CSV owns: one line per kind, with the CRD name
oc get csv "$CSV" -n openshift-operators \
  -o jsonpath='{range .spec.customresourcedefinitions.owned[*]}{.kind}{" "}{.name}{"\n"}{end}'
# The same CRDs, created in the cluster by the InstallPlan
oc get crd | grep grafana.integreatly.org

3. Where does the operator run? Which namespaces does it watch?

oc get deployments,pods -n openshift-operators
oc get operatorgroup -n openshift-operators -o yaml

4. Run oc get csv in your own namespace. Why does the Grafana Operator CSV appear there?

oc get csv -n "$NS"
oc get csv "$CSV" -n "$NS" -o jsonpath='{.status.reason}{"\n"}'

oc get clusterroles | grep grafanas.grafana.integreatly.org

Exercise 2 — Create a Grafana instance

Create a Grafana custom resource in your namespace, from the console or from the CLI.

From the console: Ecosystem > Installed Operators, select your project grafana-<id> at the top (not openshift-operators), Grafana Operator > Grafana > Create Grafana. Switch to the YAML view, replace its content with the YAML below and click Create. The Form view fails with Required value for field "spec.route.spec.to.name".

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: grafana
spec:
  config:
    log:
      mode: console
  route:
    spec: {}

From the CLI, the same YAML:

cat <<EOF | oc apply -n "$NS" -f -
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: grafana
spec:
  config:
    log:
      mode: console
  route:
    spec: {}
EOF

1. Watch the status of the instance. Which objects did the operator create?

oc get grafana grafana -n "$NS" -w
oc get deployment,service,route,configmap,secret,serviceaccount -n "$NS"
oc get deployment grafana-deployment -n "$NS" \
  -o jsonpath='{.metadata.ownerReferences[*].kind}{"/"}{.metadata.ownerReferences[*].name}{"\n"}'

2. Retrieve the admin credentials and the URL of Grafana, then log in.

oc get secret grafana-admin-credentials -n "$NS" -o jsonpath='{.data.GF_SECURITY_ADMIN_USER}' | base64 -d; echo
oc get secret grafana-admin-credentials -n "$NS" -o jsonpath='{.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d; echo
oc get route grafana-route -n "$NS" -o jsonpath='{.spec.host}{"\n"}'

curl -sk "https://$(oc get route grafana-route -n "$NS" -o jsonpath='{.spec.host}')/api/health"

Exercise 3 — Reconciliation

Delete the Grafana Deployment, then watch the Deployments of your namespace. What happens, and why?

oc delete deployment grafana-deployment -n "$NS"
oc get deployments -n "$NS" -w

Exercise 4 — Datasource and dashboard as code

Create a datasource and a dashboard for your instance:

cat <<EOF | oc apply -n "$NS" -f -
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: testdata
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  datasource:
    name: TestData
    type: grafana-testdata-datasource
    access: proxy
    isDefault: true
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: hello
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  json: |
    {
      "title": "Hello $USER",
      "panels": [
        {
          "type": "timeseries",
          "title": "Random walk",
          "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
          "targets": [{"refId": "A", "scenarioId": "random_walk"}]
        }
      ]
    }
EOF

1. Check that the dashboard appears in your Grafana. How does the operator know which Grafana instance should receive it?

oc get grafanadatasources,grafanadashboards -n "$NS"

HOST=$(oc get route grafana-route -n "$NS" -o jsonpath='{.spec.host}')
GF_USER=$(oc get secret grafana-admin-credentials -n "$NS" -o jsonpath='{.data.GF_SECURITY_ADMIN_USER}' | base64 -d)
GF_PASS=$(oc get secret grafana-admin-credentials -n "$NS" -o jsonpath='{.data.GF_SECURITY_ADMIN_PASSWORD}' | base64 -d)
curl -sk -u "$GF_USER:$GF_PASS" "https://$HOST/api/search?type=dash-db"

Exercise 5 — Uninstalling an operator

Do not run it: what would happen to your Grafana instance if the trainer uninstalled the Grafana Operator from the console?

Cleanup

oc delete project "$NS"