5.2 Tasks: Prometheus Operator

5.2.1: Prometheus storage

By default, the Prometheus operator stack does not persist the data of the deployed monitoring stack. Therefore, any pod restart would result in a reset of all data. Let’s learn about persistence for Prometheus.

Task description:

• See this example of how to configure storage for Prometheus

To get the current values, either look at the file in git or use kubectl -n $USER-monitoring get prometheus prometheus -oyaml.

...
spec:
  ...
  securityContext:
    fsGroup: 65534
  storage:
    disableMountSubPath: true
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 10Gi
  ...

5.2.2: Prometheus Retention

By default, the Prometheus operator stack will set the retention of your metrics to 24h. When metrics are persisted in Prometheus we can define the retention. Read about retention operational-aspects for options to manage retention.

Task description:

• Read the Operator’s documentation about retention and size based retention.

...
spec:
  ...
  retention: 2d
  retentionSize: 9GB
  ...

5.2.3: (Optional) Configure storage and retention in your Prometheus instance

Since we don’t always trust written documentation without testing it ourselves, we are going to configure storage and retention in our <user>-monitoring namespace.

The Prometheus resource is provided by the user-monitoring chart in your git repository. To edit the resource we need to edit the file charts/user-monitoring/templates/_user-prometheus.yaml.

Enable storage:

As we have read in the documentation, we can add the following properties to the Prometheus resource:

...
spec:
  ...
  securityContext:
    fsGroup: 65534
  storage:
    disableMountSubPath: true
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 10Gi
  ...

Configure retention:

To configure the retention and retention limits we can add the following properties to the Prometheus resource:

...
spec:
  ...
  retention: 2d
  retentionSize: 9GB
  ...

Verify the changes:

Commit and push the changes to your git repository and let ArgoCD sync your application. As soon as the pod is up and running again, we can verify that the PVC was created and bound with:

kubectl -n $USER-monitoring get pvc

Check if the volume is available inside the pod with running df -h /prometheus inside the first Prometheus pod.

kubectl -n $USER-monitoring exec prometheus-prometheus-0 -c prometheus -- df -h /prometheus

Verifying the retention config can be done by inspecting the Prometheus pod’s arguments:

kubectl -n $USER-monitoring describe pods prometheus-prometheus-0

The output should contain the following lines:

...
Containers:
  ...
  prometheus:
    ...
    Args:
      ...
      --storage.tsdb.retention.size=9GB
      --storage.tsdb.retention.time=2d
      ...