2.2 Tasks: Explore metrics
In this lab you are going to explore various metrics, which your Prometheus server is collecting.
Task 2.2.1: Prometheus/Thanos web UI
As we have now attached the Thanos Querier to your own Prometheus as well as the infrastructure Prometheus, we will use its UI to have a global view.
Get a feel for how to use the Thanos Querier web UI. Open the web UI and navigate to the Graph menu (right on top in the grey navigation bar next to Stores).
Hints
Let’s start and find a memory related metric. The best way to start is by typing node_memory in the expression bar.
Note
As soon as you start typing a dropdown with matching metrics is shown.
Select a metric such as node_memory_MemFree_bytes and click the Execute button.
The result of your first Query will be available under the two tabs:
- Table
- Graph
Explore those two views on your results. Shrink the time range in the Graph tab.
Task 2.2.2: Metric Prometheus server version
Prometheus collects its own metrics, so information such as the current build version of your Prometheus server is displayed as a metric.
Let’s find a metric that shows you the version of your Prometheus server.
Hints
Start typing prometheus_... in the expression browser, choose the prometheus_build_info metric and click the Execute Button.
Something similar to the following will be displayed
Sice we have two Prometheus Servers connected to our querier, there will be two results
prometheus_build_info{branch="HEAD", container="prometheus", endpoint="http-web", goarch="amd64", goos="linux", goversion="go1.21.0", instance="10.244.18.50:9090", job="kube-prometheus-stack-prometheus", namespace="monitoring", pod="prometheus-kube-prometheus-stack-prometheus-0", prometheus="monitoring/kube-prometheus-stack-prometheus", revision="efa34a5840661c29c2e362efa76bc3a70dccb335", service="kube-prometheus-stack-prometheus", tags="netgo,builtinassets,stringlabels", version="2.47.0"}
prometheus_build_info{branch="HEAD", container="prometheus", endpoint="web", goarch="amd64", goos="linux", goversion="go1.21.0", instance="10.244.12.41:9090", job="prometheus-operated", namespace="user6-monitoring", pod="prometheus-prometheus-0", prometheus="monitoring/kube-prometheus-stack-prometheus", revision="efa34a5840661c29c2e362efa76bc3a70dccb335", service="prometheus-operated", tags="netgo,builtinassets,stringlabels", version="2.47.0"}
The actual Version of your Prometheus Server will be available as label version
{version="2.47.0"}
Task 2.2.3: Metric TCP sockets
Let’s explore a node exporter metric in this lab.
- Find a metric that shows you the number of TCP sockets in use
- Display the number 5 minutes ago
- Display the numbers in a graph over the last 15 minutes
Hints
The node exporter metrics are all available in the node namespace
.
The number of TCP sockets in use are available in the following metric.
node_sockstat_TCP_inuse
If you want to display the value 5 minutes ago, you’ll have to add the correct timestamp in the Evaluation time field.
Switch to the Graph tab and change the value of the timepicker from 1h to 15m to display the graph over the last 15 minutes.
Task 2.2.4: Metric network interfaces
Most virtual Linux machines nowadays have network interfaces. The node exporter you have enabled and configured in the previous lab also exposes metrics about network components.
Show all disk devices where the device name starts with sd
Hints
The network interfaces are available in the following series:
node_network_info
The result includes all sorts of network interface. If you need to filter the result by a label you will have to alter your query:
node_disk_filesystem_info{device="sdc"}
But this will only show results for the exact sdc device. The Task was to show all interfaces that start with sd.
In this case we have to use Time series Selectors to create a matching filter:
node_disk_filesystem_info{device=~"sd.*"}
There will be a lot more about queries and filtering in the next Labs