3.1 Tasks: Grafana intro

Task 3.1.1: Install Grafana

Similar to the basic setup, we are just going to update our configuration of the ArgoCD application to install and create our Grafana instance. Update your monitoring application (charts/user-monitoring/values.yaml) and update the grafana.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
  enabled: true
# thanos-query
query:
  enabled: true
# grafana
grafana:
  enabled: true
# blackboxexporter
blackboxexporter:
  enabled: false
# pushgateway
pushgateway:
  enabled: false
# alertmanager
alertmanager:
  enabled: false
# thanos-ruler
ruler:
  enabled: false

Verify the installation and sync process in the ArgoCD UI . When the application is synchronized successfully navigate to your freshly created Grafana instance: https://<user>-grafana.training.cluster.acend.ch . Use the admin account to login in. Skip the password change.

Task 3.1.2: Add additional datasource to Grafana

By default, our setup adds the grafana datasource thanos-querier. This will allow us to use the combined view of metrics in the future to create dashbaords based on this data.

Navigate to the Datasouce Section in your Grafana installation and verify whether the datasources exists or not.

https://<user>-grafana.training.cluster.acend.ch/connections/datasources

Eventhough the thanos-querier datasource will be the main datasource to use in our lab, we’re going to add an additional datasource to Grafana, for you to understand how that can be configured and to have your <user>-monitoring prometheus server dircetly accessible in grafana as well.

Datasources can be added via config files to Grafana, in our case this will be handled with a secret containing the datasources in the following format:

apiVersion: 1

datasources:
  - name: Graphite
    url: http://localhost:$PORT
    user: $USER
    secureJsonData:
      password: $PASSWORD

In our example the datasources are already handled by the Helm Chart. Update your ArgoCD application and add the datasource config block to the values.yaml like the following:

# grafana
grafana:
  enabled: true
  datasources:
  - name: prometheus
    access: proxy
    editable: false
    type: prometheus
    url: http://prometheus-operated:9090

Commit your changes in the ArgoCD application to your git repository and let it synchronize.

Again go to https://<user>-grafana.training.cluster.acend.ch/connections/datasources and check for the newly added datasource.

kubectl -n $USER-monitoring get pod
kubectl -n $USER-monitoring delete pod <grafana-pod>

Task 3.1.3: Configure Prometheus to scrape Grafana metrics

This is repetition. Grafana instruments the Prometheus client library and provides a variety of metrics at the /metrics endpoint: http://<user>-grafana.training.cluster.acend.ch/metrics

This endpoint can be configured as target in prometheus, with the same methode like we did with our example application.

We simply configure a Service Monitor, which tells the prometheus server where to scrape the metrics from.

The Grafana Service Monitor has already been deployed together with grafana itself.

kubectl -n $USER-monitoring get servicemonitor grafana-monitor -oyaml

Check if the Grafana instance appears in the targets section of Prometheus (http://<user>-prometheus.training.cluster.acend.ch/targets ). In addition you can use the following query to show list all metrics of the new target:

{job="grafana"}

Task 3.1.4: Use datasources in grafana

Since we have our datasouces now configured in Grafana, we can use the explore tab in grafana, to check, whether it’s working correctly.

• Open your grafana https://<user>-grafana.training.cluster.acend.ch
• Use the navigation to open the Explore site.
• Select the prometheus datasource
• Switch to code mode in your query input section
• Execute the following query

python_info

The result will be the same like the one from Lab 1.

As we are most likely always more interested in the global view, we are again going to use the thanos-querier datasource instead of the prometheus one.

• Change the datasource to thanos-querier
• Execute the same query again.

Again the same result.

But, since the querier now combines the metrics from our two prometheus stacks (Cluster infrastructure and user-monitoring) we now have the possibility to also query cluster metrics:

kube_node_info

Which will give you info about our underlying kubernetes cluster nodes. Such metrics are only available on the thanos-querier datasource.