4.3 Tasks: Exporter as a sidecar
Task 4.3.1: Deploy a database and use a sidecar container to expose metric
As we’ve learned in Lab 4 - Prometheus exporters when applications do not expose metrics in the Prometheus format, there are a lot of exporters available to convert metrics into the correct format. In Kubernetes this is often done by deploying so called sidecar containers along with the actual application.
This lab demonstrates that with a mariadb database. In the first step, we are going to install a plain mariadb with a Secret
(username password to access the database), a Service
and the Deployment
.
Create the following three files with the corresponding filenames and content:
user-demo/mariadb-secret.yaml:
---
apiVersion: v1
kind: Secret
metadata:
name: mariadb
labels:
app: mariadb
data:
database-name: YWNlbmRleGFtcGxlZGI=
database-password: bXlzcWxwYXNzd29yZA==
database-root-password: bXlzcWxyb290cGFzc3dvcmQ=
database-user: YWNlbmQtdXNlcg==
user-demo/mariadb-deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb
labels:
app: mariadb
spec:
selector:
matchLabels:
app: mariadb
strategy:
type: Recreate
template:
metadata:
labels:
app: mariadb
spec:
containers:
- image: mariadb:10.5
imagePullPolicy: IfNotPresent
name: mariadb
args:
- --ignore-db-dir=lost+found
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
key: database-user
name: mariadb
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
key: database-password
name: mariadb
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
key: database-root-password
name: mariadb
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
key: database-name
name: mariadb
livenessProbe:
tcpSocket:
port: 3306
ports:
- containerPort: 3306
name: mariadb
user-demo/mariadb-service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: mariadb
labels:
app: mariadb
prometheus-monitoring: 'true'
spec:
ports:
- name: mariadb
port: 3306
protocol: TCP
targetPort: 3306
selector:
app: mariadb
Add, commit and push the changes to your git repository and ArgoCD will deploy the mariadb.
This will create a Secret (username password to access the database), a Service and the Deployment .
MariaDB is not able to expose prometheus metrics out of the box, we need to deploy the mariadb exporter from https://quay.io/prometheus/mysqld-exporter/ as a sidecar container
For that we must alter the existing MariaDB deployment and service definition to contain the side car.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mariadb
labels:
app: mariadb
spec:
selector:
matchLabels:
app: mariadb
strategy:
type: Recreate
template:
metadata:
labels:
app: mariadb
spec:
containers:
- image: quay.io/prometheus/mysqld-exporter:v0.14.0
imagePullPolicy: IfNotPresent
name: mariadb-exporter
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
key: database-user
name: mariadb
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
key: database-password
name: mariadb
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
key: database-root-password
name: mariadb
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
key: database-name
name: mariadb
- name: DATA_SOURCE_NAME
value: $(MYSQL_USER):$(MYSQL_PASSWORD)@(127.0.0.1:3306)/$(MYSQL_DATABASE)
ports:
- containerPort: 9104
name: mariadb-exp
- image: mariadb:10.5
name: mariadb
args:
- --ignore-db-dir=lost+found
env:
- name: MYSQL_USER
valueFrom:
secretKeyRef:
key: database-user
name: mariadb
- name: MYSQL_PASSWORD
valueFrom:
secretKeyRef:
key: database-password
name: mariadb
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
key: database-root-password
name: mariadb
- name: MYSQL_DATABASE
valueFrom:
secretKeyRef:
key: database-name
name: mariadb
livenessProbe:
tcpSocket:
port: 3306
ports:
- containerPort: 3306
name: mariadb
---
apiVersion: v1
kind: Service
metadata:
name: mariadb
labels:
app: mariadb
prometheus-monitoring: 'true'
spec:
ports:
- name: mariadb
port: 3306
protocol: TCP
targetPort: 3306
- name: mariadb-exp
port: 9104
protocol: TCP
targetPort: 9104
selector:
app: mariadbThen we also need to create a new ServiceMonitor (user-demo/mariadb-servicemonitor.yaml) to instruct Prometheus to scrape the sidecar container on the given port:
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
app.kubernetes.io/name: mariadb
name: mariadb
spec:
endpoints:
- interval: 60s
port: mariadb-exp
scheme: http
path: /metrics
selector:
matchLabels:
prometheus-monitoring: 'true'
Explore the changes we did to the resources and verify that the target gets scraped in the Prometheus user interface
. Target name: serviceMonitor/<user>/mariadb/0 (It may take up to two minutes for Prometheus to load the new configuration and
scrape the metrics).
Task 4.3.2: (Optional) Explore mariadb Metrics
Explore the newly added metrics using the following PromQL Query:
{job="mariadb"}
Import the mariadb dashboard and explore how those metrics can be used.