4.1 Tasks: Blackbox exporter

Task 4.1.1: Install Blackbox exporter

Similar to the basic setup, we are just going to update our configuration of the ArgoCD application to install the Blackbox exporter. Update your monitoring application (charts/user-monitoring/values.yaml) and update the blackboxexporter.enabled flag to true:

charts/user-monitoring/values.yaml:

user: <user> # Replace me
# prometheus
prometheus:
  enabled: true
# thanos-query
query:
  enabled: true
# grafana
grafana:
  enabled: true
  datasources:
  - name: prometheus
    access: proxy
    editable: false
    type: prometheus
    url: http://prometheus-operated:9090
# blackboxexporter
blackboxexporter:
  enabled: true
# pushgateway
pushgateway:
  enabled: false
# alertmanager
alertmanager:
  enabled: false
# thanos-ruler
ruler:
  enabled: false

Commit and push the changes.

Verify the installation and sync process in the ArgoCD UI . Or execute the following command:

kubectl -n $USER-monitoring get pod

Task 4.1.1: Add a blackbox target

We will use the blackbox exporter to create a new probe which accepts a 2xx return code as a valid http return code. This will return the probe_success metric from the blackbox exporter with the value 1, if the http status code is 2xx.

Task description:

Create a probe (user-demo/training_blackbox_target.yaml) which uses the HTTP prober and expects a 2xx return code as a valid status code
Define https://www.acend.ch as a single static target, which the blackbox should probe

Hints

To configure the blackbox exporter you have to add the following file user-demo/training_blackbox_target.yaml to your directory, commit and push the changes:

apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: acend-2xx
spec:
  module: http_2xx
  prober:
    url: blackbox:9115
  targets:
    staticConfig:
      static:
      - https://www.acend.ch

The Prometheus server translates this Probe Custom Resource into a prometheus target, which gets scraped accordingly to its scrape interval.

Url: http://<blackboxexportertservice>?target=https://www.acend.ch&module=http_2xx

Check the prometheus target configuration in the Prometheus UI .

Open a new Terminal and execute the following command

kubectl -n $USER-monitoring port-forward service/blackbox 9115:9115

This will open a port-forward to the blackbox exporter.

Switch back to the initial terminal and execute this command

curl localhost:9115/probe?target=https://www.acend.ch&module=http_2xx

The blackbox exporter checks the given target url and returns metrics, the metric probe_success metric should have the value 1.

...
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 1
...

Task 4.1.2: Query blackbox metrics

Let’s now create a query which selects all metrics belonging to the blackbox exporter target https://www.acend.ch and display them in the Thanos Querier UI .

Hints

We can select all metrics for the target with the following query:

{instance="https://www.acend.ch"}

Warning

In the list of metrics you will find one metric with the name up. In the case of a multi-target exporter such as the blackbox exporter this metric will always be up as long as Prometheus is able to successfully scrape the exporter even if the actual target (website, TCP service, etc.) is down. To monitor the state of the targets always use the probe_success metric.

Task 4.1.3 (optional): Add a protocol label to your blackbox target

Add the new label protocol to every blackbox exporter target by updating the relabel config. The new label should contain the protocol (HTTP or HTTPS) extracted from the target URL.

Hints

To configure the blackbox exporter you have to updates the following file training_blackbox_target.yaml in your directory:

apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: acend-2xx
spec:
  module: http_2xx
  prober:
    url: blackbox:9115
  targets:
    staticConfig:
      static:
      - https://www.acend.ch
  metricRelabelings:
  - sourceLabels: [instance] #1
    targetLabel: protocol #2
    regex: '^(.+):.+' #3
    replacement: $1 #4

1: Use the value from the label instance. This label contains all targets defined at .spec.targets.staticConfig.static
2: We will call the new label protocol
3: Capture the first part of your url until :. In our case https from https://acend.ch/
4: Replace target_label value with the regex match from source_labels value

Task 4.1.4 (optional): PromQL Query, ssl certificate expire

The blackbox exporter provides a lot of metrics. One very common usecase is, to create a dashboard or alert for expiring SSL certificates.

Wrtie a query, that returns the days until the ssl certificate of https://www.acend.ch expires.

Info

Hint: Use the probe_ssl_earliest_cert_expiry as a starting point. This metric returns the timestamp in seconds, when the certificate will expire. The result of the time() function, is the current timestamp.

Hints

# solution
(probe_ssl_earliest_cert_expiry - time()) / (3600 *24)

Last modified April 29, 2026: Merge 16c2cf49af91125a95fbffd0197a473293446d89 into bab7e50cc1efa022e59121d708c7de54c6d4d4ce (1ef1573)