7.2 Tasks: Alertmanager
Task 7.2.1: Configure Slack as alert receiver
As part of our lab setup we already configured an alert receiver for Alertmanager alerts.
We deployed the Mail catcher , which is a very simple component. We can send Emails to the server, and those are then displayed in the Web UI. The Emails are not send anywhere, and this setup should only be used for demo purposes.
When we enabled the alertmanager configuration in the charts/user-monitoring/values.yaml we also deployed the following AlertmanagerConfig Custom Resource
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
name: <user>-mailcatcher
labels:
alertmanagerConfig: <user>-alertmanager
spec:
route:
groupBy: ['job']
groupWait: 30s
groupInterval: 5m
repeatInterval: 12h
receiver: 'mailcatcher'
receivers:
- name: 'mailcatcher'
emailConfigs:
- to: alert@localhost
from: prometheus-operator@localhost
smarthost: mailcatcher:1025
requireTLS: false
When an alert is firing it will send an email to the Mail catcher.
Task 7.2.2: Send a test alert
In this task you can use the amtool command to send a test alert.
To send a test alert with the labels alertname=Up and node=bar you can simply execute the following command.
kubectl --namespace $USER-monitoring exec -it statefulsets/alertmanager-$USER-alertmanager -c alertmanager -- sh
amtool alert add --alertmanager.url=http://localhost:9093 alertname=Up node=bar
Check in the Alertmanager UI if you see the test alert with the correct labels set. Info: this will not yet send an email to the Mail catcher. You’ll find out why in the end of this lab.
Task 7.2.3: Show the routing tree
Show routing tree:
kubectl --namespace $USER-monitoring exec -it statefulsets/alertmanager-$USER-alertmanager -c alertmanager -- sh
amtool config routes --alertmanager.url=http://localhost:9093
Depending on the configured receivers your output might vary.
If you only configured the email receiver, the output will look similar to this:
kubectl --namespace $USER-monitoring exec -it statefulsets/alertmanager-$USER-alertmanager -c alertmanager -- sh
amtool config routes --config.file /etc/alertmanager/alertmanager.yml
Routing tree:
.
└── default-route receiver: default
├── {namespace="<user>-monitoring"} continue: true receiver: <user>-monitoring/<user>-mailcatcher/mailcatcher
Task 7.2.5: Test your alert receivers
Add a test alert and check if alert has been sent to the Mail catcher . It can take up to 5 minutes as the alarms are grouped together based on the group_interval .
kubectl --namespace $USER-monitoring exec -it statefulsets/alertmanager-$USER-alertmanager -c alertmanager -- sh
amtool alert add --alertmanager.url=http://localhost:9093 alertname=snafu env=dev severity=critical
It is also advisable to validate the routing configuration against a test dataset to avoid unintended changes. With the option --verify.receivers the expected output can be specified:
kubectl --namespace $USER-monitoring exec -it statefulsets/alertmanager-$USER-alertmanager -c alertmanager -- sh
amtool config routes test --alertmanager.url=http://localhost:9093 --verify.receivers=<user>-monitoring/<user>-mailcatcher/mailcatcher env=dev severity=info
default
WARNING: Expected receivers did not match resolved receivers.
Only alerts with the namespace=<user>-monitoring will be routed to our mailcatcher.
kubectl --namespace $USER-monitoring exec -it statefulsets/alertmanager-$USER-alertmanager -c alertmanager -- sh
amtool config routes test --config.file /etc/alertmanager/alertmanager.yml --verify.receivers=<user>-monitoring/<user>-mailcatcher/mailcatcher env=dev namespace=<user>-monitoring