1.1 Tasks: Setup

Task 1.1.1: Getting Started - Web IDE

The first thing we’re going to do is to explore our lab environment and get in touch with the different components.

The namespace with the name corresponding to your username is going to be used for all the hands-on labs. And you will be using the folling tools during the lab

• Gitea Git Server: https://gitea.training.cluster.acend.ch - Login with <user> and the provided password
• Argo CD Server: https://argocd.training.cluster.acend.ch - Login via Gitea
• git - Login with <user> and the provided password
• kubectl - already logged in

Once you’re successfully logged into the web IDE open a new Terminal by hitting CTRL + SHIFT + ¨ or clicking the Menu button –> Terminal –> new Terminal and check the installed kubectlversion by executing the following command:

kubectl version --output=yaml

The Web IDE Pod consists of the following tools:

• oc
• kubectl
• kustomize
• helm
• kubectx
• kubens
• tekton cli
• argocd

The files in the home directory under /home/project are stored in a persistence volume, so please make sure to store all your persistence data in this directory.

Task 1.1.1.1: Local Workspace Directory

During the lab, you’ll be using local files (eg. YAML resources) which will be applied in your lab project.

Create a new folder for your <workspace> in your Web IDE (for example prometheus-training under /home/project/prometheus-training). Either you can create it with right-mouse-click -> New Folder or in the Web IDE terminal

mkdir prometheus-training && cd prometheus-training

In the Web IDE we set the USER environment variable to your personal <username>.

Verify that with the following command:

echo $USER

The USER variable will be used as part of the commands to make the lab experience more comfortable for you.

Clone the forked repository to your local workspace:

git clone https://$USER@gitea.training.cluster.acend.ch/$USER/prometheus-training-lab-setup.git

Change the working directory to the cloned git repository:

cd prometheus-training-lab-setup

For convenience let’s configure the git client:

git config user.name "$USER"
git config user.email "$USER@gitea.training.cluster.acend.ch"

And we also want git to store our Password for two days so that we don’t need to login every single time we push something.

git config credential.helper 'cache --timeout=172800'

Then use the following command to verify whether the git config for username and email were correctly added:

git config --local --list

Explore the cloned repository.

Task 1.1.2: Install Prometheus

As explained in the previous section, we’re going to use ArgoCD to deploy our Kubernetes resources for our lab, therefore the first thing we do is to create the ArgoCD configuration.

Configure ArgoCD correctly

In order for ArgoCD to monitor and synchronize your applications correctly, we need first to make ArgoCD aware of the applications to be deployed. ArgoCD application resources (Application) create a logical connection for ArgoCD between a git repository and a kubernetes namespace. In your cloned repository you will have already have two ArgoCD applications prepared for you (apps/user-demo.yaml and apps/user-prom-stack.yaml). The first application apps/user-demo.yaml will synchronize and deploy your user workload examples and the second application apps/user-prom-stack.yaml will be used to deploy the prometheus infrastructure resources.

Open these two files (apps/user-demo.yaml and apps/user-prom-stack.yaml) in your editor and replace all the <user> placeholders with your correct username. Safe the files and push them to your git repository:

git add .
git commit -m "Replace userid"
git push

So far, nothing has happened yet. We create a third ArgoCD application to synchronize the other two applications. In ArgoCD we call this the app-of-apps pattern . In order to do that, we create a file user-app-of-apps.yaml in the root directory of your git repository. Add the following content and replace all <user> placeholder with your user:

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: <user>-app-of-apps
  namespace: argocd
spec:
  destination:
    namespace: argocd
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: 'https://gitea.training.cluster.acend.ch/<user>/prometheus-training-lab-setup'
    path: apps/
    targetRevision: main
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - Replace=true

kubectl -n argocd create -f user-app-of-apps.yaml

Deploy Prometheus

As mentioned our Prometheus Stack will be deployed in the <user>-monitoring namespace. The instance itself will be deployed via ArgoCD with the defined application in your git repository. In order to create our prometheus instance we need to alter the configuration of the Helm chart in charts/user-monitoring. Open the charts/user-monitoring/values.yaml file in your editor and change the <user> placeholder correctly and update the value of prometheus.enabled to true.

user: <user> # Replace me
# prometheus
prometheus:
  enabled: true
# thanos-query
query:
  enabled: false
# grafana
grafana:
  enabled: false
# blackboxexporter
blackboxexporter:
  enabled: false
# pushgateway
pushgateway:
  enabled: false
# alertmanager
alertmanager:
  enabled: false
# thanos-ruler
ruler:
  enabled: false

If you are curious to see what ArgoCD will do for you, you can render the helmchart locally:

helm template charts/user-monitoring/.

When you are confident that the changes are done correctly, simply commit and push the files to your git repository and from there ArgoCD will take over and start synchronizing.

git add .
git commit -m "Enable Prometheus"
git push

Head over to the ArgoCD UI and verify that the synchronization process of your application is synced and healthy. As soon as your application is healthy and synced (green status on top) you are good to go and have your prometheus instance ready.

When all has finished syncing, you can inspect your prometheus installation in your namespace:

kubectl -n $USER-monitoring get prometheus prometheus -oyaml

Configure Prometheus

As mentioned in the introduction, configuring a Prometheus on Kubernetes can be done using the Prometheus Operator. We basically need to specify a Prometheus custome resouce and the Operator will do its work. In the prometheus custom resource’s spec block you can find various configuration options:

spec:
  enableAdminAPI: true
  evaluationInterval: 30s
  externalLabels:
    monitoring: <user>
  podMonitorNamespaceSelector:
    matchLabels:
      user: <user>
  podMonitorSelector: {}
  portName: web
  probeNamespaceSelector:
    matchLabels:
      user: <user>
  probeSelector: {}
  resources:
    requests:
      memory: 400Mi
  scrapeInterval: 60s
  serviceAccountName: prometheus-<user>
  serviceMonitorNamespaceSelector:
    matchLabels:
      user: <user>
  serviceMonitorSelector: {}

So far we do not have configured anything special yet. We instruct prometheus to automatically discover Probes, PodMonitors and ServiceMonitors in all namespaces with the label user matching your username.

Check Prometheus

Is your prometheus running? Use your browser to navigate to https://<user>-prometheus.training.cluster.acend.ch . You should now see the Prometheus web UI.

Task 1.1.3: Deploy example application

In the next section we are going to deploy our own application and try to monitor their metrics. As stated earlier the folder user-demo is automatically watched and synchronized by ArgoCD from your git repository. Deploy the Acend example Python application, which provides application metrics at /metrics by creating the following file (user-demo/deployment.yaml) in your repo:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: example-web-python
  name: example-web-python
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-web-python
  template:
    metadata:
      labels:
        app: example-web-python
    spec:
      containers:
      - image: quay.io/acend/example-web-python
        name: example-web-python
        resources:
          requests:
            memory: "32Mi"
            cpu: "10m"
          limits:
            memory: "128Mi"
            cpu: "100m"

We can simply deploy the application by pushing the resource into our git repository in the user-demo folder:

git add .
git commit -m "Deploy Demo App"
git push

Use the following command to verify whether pod example-web-python is Ready and Running in your <user> namespace. (use CTRL C to exit the command) Or you can also check in the ArgoCD UI .

kubectl -n $USER get pod -w

To access the deployed resources we also need to create a Service for the new application. Create a file (user-demo/service.yaml) with the following content:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: example-web-python
    prometheus-monitoring: 'true'
  name: example-web-python
spec:
  ports:
    - name: http
      port: 5000
      protocol: TCP
      targetPort: 5000
  selector:
    app: example-web-python
  type: ClusterIP

We can add and push the resource to our git repository to have it synchronized to our namespace:

git add .
git commit -m "Deploy Demo Service"
git push

This created a so-called Kubernetes Service which allows communication via an internal network abstraction.

kubectl -n $USER get services

Task 1.1.4: Create a ServiceMonitor

So far we did not have any interaction with Prometheus at all. Kubernetes-based Prometheus installation use PodMonitors and ServiceMonitors as a service-discovery mechanism. PodMonitors and ServiceMonitors select via labels pods / services for prometheus to scrape at a given endpoint. Check whether the application metrics are actually exposed by opening a shell within the container and curling the metrics endpoint.

# get the pod name
kubectl -n $USER get pod

# exec curl within the pod
kubectl -n $USER exec -it <pod-name> -- curl http://localhost:5000/metrics

Should result in something like:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 541.0
python_gc_objects_collected_total{generation="1"} 344.0
python_gc_objects_collected_total{generation="2"} 15.0
...

Since our newly deployed application now exposes metrics, the next thing we need to do, is to tell our Prometheus server to scrape metrics from the Kubernetes deployment. In a highly dynamic environment like Kubernetes this is done with so called Service Discovery.

Let us create a ServiceMonitor for the example application, which will configure Prometheus to scrape metrics from the example-web-python application every 30 seconds.

For this to work, you need to ensure:

• The example-web-python Service is labeled correctly and matches the labels you’ve defined in your ServiceMonitor.
• The port name in your ServiceMonitor configuration matches the port name in the Service definition.
  • hint: check with kubectl -n $USER get service example-web-python -o yaml
• Verify the target in the Prometheus user interface.

Create the following ServiceMonitor user-demo/servicemonitor.yaml in your git repository:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: example-web-python
  name: example-web-python-monitor
spec:
  endpoints:
    - interval: 60s
      port: http
      scheme: http
      path: /metrics
  selector:
    matchLabels:
      prometheus-monitoring: 'true'

Then commit and push the resource again for ArgoCD to synchronize.

git add .
git commit -m "Add ServiceMonitor"
git push

In the Prometheus UI we can check our targets to be scaped. So far no target should appear from your demo application and ServiceMonitor we just deployed. This is happening because we tell Prometheus to look for ServiceMonitors in our <user> namespace, but it is not yet allowed by the Kubernetes API to see resources in this namespace. Therefore we need to add RBAC rules to let Prometheus see and scrape these resources.

We do this again by creating a Role and RoleBinding for the Prometheus’ ServiceAccount. Create the following two files in your git repository:

user-demo/role.yaml Replace the <user> placeholder with your user:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: prometheus-<user>
rules:
- apiGroups: [""]
  resources:
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]

user-demo/rolebinding.yaml Replace the <user> placeholder with your user:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: prometheus-<user>
subjects:
- kind: ServiceAccount
  name: prometheus-<user>
  namespace: <user>-monitoring

Then again add and push these files to your git repository and let ArgoCD synchronize them for you.

git add .
git commit -m "Add role and rolebinding"
git push

Verify that the target gets scraped in the Prometheus user interface . Target name: serviceMonitor/<user>/example-web-python-monitor/0 (1/1 up) (it may take up to a minute for Prometheus to load the new configuration and scrape the metrics).

Task 1.1.5: Your first PromQL Query

Switch back to the Graph Tab in the Prometheus UI and enter the following query into the input field:

python_info

hit enter and explore the result should look similar to

python_info{endpoint="http", implementation="CPython", instance="10.244.18.95:5000", job="example-web-python", major="3", minor="11", namespace="<user>", patchlevel="5", pod="example-web-python-7c8b9984d4-z9b92", service="example-web-python", version="3.11.5"}  1