4.2 Tasks: Pushgateway
Task 4.2.1 - Install and configure Pushgateway
Update your monitoring application (charts/user-monitoring/values.yaml) and update the blackboxexporter.enabled flag to true:
charts/user-monitoring/values.yaml:
user: <user> # Replace me
# prometheus
prometheus:
enabled: true
# thanos-query
query:
enabled: true
# grafana
grafana:
enabled: true
datasources:
- name: prometheus
access: proxy
editable: false
type: prometheus
url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
enabled: true
# pushgateway
pushgateway:
enabled: true
# alertmanager
alertmanager:
enabled: false
# thanos-ruler
ruler:
enabled: false
Commit and push the changes and verify whether the deployment worked correctly.
Task 4.2.2 - Push metrics to Pushgateway
In this task you’re going to push metrics to the Pushgateway. This is what you would normally do, after a cronjob has completed successfully.
In order to push
metrics to the Pushgateway, you can simply send an HTTP POST or PUT request, with the actual metric we want to push as content.
When pushing metrics to the Pushgateway, you always have to specify the job, therefore the URL Path looks like this:
http://localhost:9091/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}
If we want to push the metric prometheus_training_labs_completed_total with the value 4 and the job prometheus_training to the Pushgateway, we can do that by creating the following Kubernetes Job:
kubectl -n $USER-monitoring create job --image=quay.io/acend/example-web-python pushgw-example1 -- \
sh -c "echo 'prometheus_training_labs_completed_total 3' | curl --data-binary @- http://pushgateway:9091/metrics/job/prometheus_training"
Command Explanation
If you are not very familiar with kubectl create job. The above command does the following:
kubectl -n ... create jobcreates an adhoc kubernetes job--image=specifies, which image the container will use. We will use the toolkit container because it provides bash and curl.pushgw-example1is the name of the jobbash -c "..."is the command, the job should execute
Verify the metric in the Thanos Querier web UI
. It may take up to 30s ( Depending on the scrape_interval) to be available in Prometheus.
Push
the following metric (notice the instance label) to the Pushgateway and make sure the metric gets scraped by Prometheus
# TYPE some_metric_total counter
# HELP This is just an example metric.
some_metric_total{job="prometheus_training",instance="myinstance"} 42
Hints
To push a metric to the Pushgateway, which will then be scraped by Prometheus, we can simply create the following job. Note the actual content of the HTTP request, is exactly the metric we want Prometheus to scrape.
Execute the following command to push the metric to your Pushgateway:
kubectl -n $USER-monitoring create job --image=quay.io/acend/example-web-python pushgw-example2 -- \
sh -c "cat <<EOF | curl --data-binary @- http://pushgateway:9091/metrics/job/prometheus_training/instance/myinstance
# TYPE some_metric_total counter
# HELP This is just an example metric.
some_metric_total 42
EOF"
Command Explanation
If you are not very familiar with the Linux shell, the above command does the following:
- the
catcommand reads the actual metric and pipes it tostdin - curl sends a HTTP POST request to the URL http://pushgateway:9091/metrics/job/prometheus_training/instance/myinstance
with the –data-binary parameter set to
stdin(the actual metric)
Verify the metric in the Thanos Querier web UI
. It may take up to 30s (depending on the scrape_interval) to be available in Prometheus.
Task 4.2.3 - Delete Pushgateway metrics
By sending HTTP delete requests to the same endpoint, we can delete metrics from the Pushgateway.
Note
Metrics pushed to the Pushgateway are not automatically purged until you manually delete them via the API or the process restarts. If you persist the metrics with--persistence.file, you should ensure that you have set up a job that cleans up the metrics on a regular basis.According to the official Pushgateway documentation you can delete either metrics for specific label combinations (exact match required) or all metrics.
Delete the pushed metrics from the Pushgateway.
Hints
To delete the metrics for the job prometheus_training, you can simply execute the following command:
kubectl -n $USER-monitoring create job --image=quay.io/acend/example-web-python pushgw-delete -- \
curl -X DELETE http://pushgateway:9091/metrics/job/prometheus_training
Note
This will delete metrics with the label set {job="prometheus_training"} but not {job="prometheus_training",another_label="value"} since the delete methode requires an exact label match.
The Pushgateway pod has no persistence, so you can delete all metrics stored in Pushgateway by deleting the pod.
kubectl -n $USER-monitoring delete pod -l app.kubernetes.io/name=pushgateway
Remove the created examples jobs.
kubectl -n $USER-monitoring delete jobs pushgw-delete pushgw-example1 pushgw-example2