Setting up a Prometheus monitoring stack in Kubernetes, utilising helm.

Craig Godden-Payne

Nov 1, 2020·10 min read

Pre-requisites

In this article, I will be working with the following software, it makes sense to have these pre-installed before continuing.

• minikube - minikube is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes
• kubectl - The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters.
• k9s - K9s provides a terminal UI to interact with your Kubernetes clusters.
• helm - helm Charts help you define, install, and upgrade even the most complex Kubernetes configurations

You can install all by using brew if on OSX, or check out the website for detailed installation instructions.

Creating a Namespace

Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.

Having a namespace is going to make your life much simpler, when it comes to adding applications to be monitored, as they will be segregated.

So to do this, first of all, create a file called namespace.yaml

Whether you keep this separate or with your application is up to you. Both use cases are common practice, I prefer to keep all namespace definitions separate to application stacks, so when you see examples below, it will be separated out.

Here's how I intend to layout my code:

.
├── application
├── cluster
│   └── namespace.yaml
└── monitoring

Cluster level config is stored under cluster
Monitoring level config under monitoring
Applications config under application

Running in changes

In order to create the namespace (and all other configs going forward), we need to use the kubectl command.

Since the structure is segregated into directories, we need to cd to the cluster directory, and run the following command:

$ cd cluster$ kubectl apply -f *.yaml
namespace/monitoring created$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   355d
kube-node-lease   Active   355d
kube-public       Active   355d
kube-system       Active   355d
monitoring        Active   65s

You should now have a running namespace

Creating a Helm Template

We are going to use helm to help us with the prometheus stack, it will save us a huge amount of time in the long run, as it will template the app for us, and make it more manageable.

Run the following command:

$ helm create prometheus

This will create a new helm chart, and will have a structure similar to this:

.
├── Chart.yaml
├── charts
├── templates
│   ├── NOTES.txt
│   ├── _helpers.tpl
│   ├── deployment.yaml
│   ├── hpa.yaml
│   ├── ingress.yaml
│   ├── service.yaml
│   ├── serviceaccount.yaml
│   └── tests
│       └── test-connection.yaml
└── values.yaml

Chart.yaml and values.yaml will define what the chart is, and what values will be in it at deployment.

From here, we can deploy so you can test the help chart works. By default it will run nginx, and you should be able to hit one deployed. (I have shown in debug mode so you can see all the resources being created)

$ helm upgrade \
  prometheus . \
  -n monitoring  \
  --install \
  --atomic \
  --debug="true" \
  --dry-run="false"history.go:53: [debug] getting history for release prometheus
Release "prometheus" does not exist. Installing it now.
install.go:172: [debug] Original chart version: ""
install.go:189: [debug] CHART PATH: /Users/craiggoddenpayne/Dropbox/Work/BeardyDigital/code/k8s-metrics-stack/prometheusclient.go:109: [debug] creating 4 resource(s)
wait.go:53: [debug] beginning wait for 4 resources with timeout of 5m0s
wait.go:225: [debug] Deployment is not ready: monitoring/prometheus. 0 out of 1 expected pods are ready
NAME: prometheus
LAST DEPLOYED: Sat Oct 31 12:37:45 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
USER-SUPPLIED VALUES:
{}COMPUTED VALUES:
affinity: {}
autoscaling:
  enabled: true
  maxReplicas: 3
  minReplicas: 1
  targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
  repository: nginx
imagePullSecrets: []
ingress:
  annotations: {}
  enabled: false
  hosts:
  - host: chart-example.local
    paths: []
  tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 1
resources: {}
securityContext: null
service:
  port: 80
  type: ClusterIP
serviceAccount:
  annotations: {}
  create: true
  name: ""
tolerations: []HOOKS:
---
# Source: prometheus/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: "prometheus-test-connection"
  labels:
    helm.sh/chart: prometheus-0.1.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
    "helm.sh/hook": test
spec:
  containers:
    - name: wget
      image: busybox
      command: ['wget']
      args: ['prometheus:80']
  restartPolicy: Never
MANIFEST:
---
# Source: prometheus/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-0.1.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
---
# Source: prometheus/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-0.1.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
---
# Source: prometheus/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-0.1.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/instance: prometheus
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus
        app.kubernetes.io/instance: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        {}
      containers:
        - name: prometheus
          securityContext:
            null
          image: "nginx:1.16.0"
          imagePullPolicy:
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {}
---
# Source: prometheus/templates/hpa.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-0.1.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "1.16.0"
    app.kubernetes.io/managed-by: Helm
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: prometheus
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace monitoring $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace monitoring port-forward $POD_NAME 8080:$CONTAINER_PORT

To hit the pod, you will need to expose it to the host, and to do this you will need to get the pod name by running:

$ kubectl get pod -n monitoringNAME                          READY   STATUS    RESTARTS   AGE
prometheus-54d964dbb7-qhztr   1/1     Running   0          3m33s

Then use the following command to port forward to the host:

kubectl port-forward -n monitoring pods/prometheus-54d964dbb7-qhztr 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080

So now, container port 80, is mapped to port 8080 on your local machine.

Now, hit that in a browser

Updating the Helm Chart to setup Prometheus

If we have a nosy around in the values.yaml file, you will see where nginx is referenced.

We should be able to change this, to point to a Prometheus image.

Update this in the values.yaml file

image:
  repository: docker.io/prom/prometheus

And make sure to update the application version in Chart.yaml

appVersion: v2.20.0

Next update the service port from 80, to 9090

service:
  type: ClusterIP
  port: 9090

And make sure to updates the probes in deployment.yaml to look at port 9090, or the container will keep cycling

ports:
  - name: http
    containerPort: 9090
    protocol: TCP

Run the deploy again to make sure that there are no errors:

$ helm upgrade \
  prometheus . \
  -n monitoring  \
  --install \
  --atomic \
  --debug="true" \
  --dry-run="false"history.go:53: [debug] getting history for release prometheus
Release "prometheus" does not exist. Installing it now.
install.go:172: [debug] Original chart version: ""
install.go:189: [debug] CHART PATH: /Users/craiggoddenpayne/Dropbox/Work/BeardyDigital/code/k8s-metrics-stack/prometheusclient.go:109: [debug] creating 4 resource(s)
wait.go:53: [debug] beginning wait for 4 resources with timeout of 5m0s
wait.go:225: [debug] Deployment is not ready: monitoring/prometheus. 0 out of 1 expected pods are ready
NAME: prometheus
LAST DEPLOYED: Sat Oct 31 14:14:20 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
USER-SUPPLIED VALUES:
{}COMPUTED VALUES:
affinity: {}
autoscaling:
  enabled: true
  maxReplicas: 3
  minReplicas: 1
  targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
  repository: docker.io/prom/prometheus
imagePullSecrets: []
ingress:
  annotations: {}
  enabled: false
  hosts:
  - host: chart-example.local
    paths: []
  tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 1
resources: {}
securityContext: null
service:
  port: 9090
  type: ClusterIP
serviceAccount:
  annotations: {}
  create: true
  name: ""
tolerations: []HOOKS:
---
# Source: prometheus/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: "prometheus-test-connection"
  labels:
    helm.sh/chart: prometheus-1.0.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "v2.20.0"
    app.kubernetes.io/managed-by: Helm
  annotations:
    "helm.sh/hook": test
spec:
  containers:
    - name: wget
      image: busybox
      command: ['wget']
      args: ['prometheus:9090']
  restartPolicy: Never
MANIFEST:
---
# Source: prometheus/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-1.0.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "v2.20.0"
    app.kubernetes.io/managed-by: Helm
---
# Source: prometheus/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-1.0.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "v2.20.0"
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
    - port: 9090
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
---
# Source: prometheus/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-1.0.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "v2.20.0"
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/instance: prometheus
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus
        app.kubernetes.io/instance: prometheus
    spec:
      serviceAccountName: prometheus
      securityContext:
        {}
      containers:
        - name: prometheus
          securityContext:
            null
          image: "docker.io/prom/prometheus:v2.20.0"
          imagePullPolicy:
          ports:
            - name: http
              containerPort: 9090
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {}
---
# Source: prometheus/templates/hpa.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: prometheus
  labels:
    helm.sh/chart: prometheus-1.0.0
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/version: "v2.20.0"
    app.kubernetes.io/managed-by: Helm
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: prometheus
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80

Awesome, and lets run a port forward to check that it’s actually working:

$ kubectl get pods -n monitoring
NAME                          READY   STATUS    RESTARTS   AGE
prometheus-8644c59db5-w6c8k   1/1     Running   0          106s$ kubectl port-forward -n monitoring pods/prometheus-8644c59db5-w6c8k 8080:9090Forwarding from 127.0.0.1:8080 -> 9090
Forwarding from [::1]:8080 -> 9090

Cool so now we have a running prometheus pod, defined within a helm chart.

Reconfiguring Prometheus

So having a default prometheus instance is great, but we need to add customisations to make it useful.

The first thing we should look at, is the configuration file.

At the moment it is using the default value supplied from within the docker image.

Lets swap this out with a config map.

Create a file called prometheus-config-map.yaml

Add the following contents:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-config
data:
  prometheus.yml: |
    global:
      scrape_interval:     15s
      evaluation_interval: 15s

    scrape_configs:
      - job_name: prometheus
        metrics_path: /metrics
        scheme: http

        static_configs:
        - targets: ['localhost:9090']

Now we need to tell the docker container to swap out the volume and file, with the above config map, so the container sees our config rather than the default config.

You can do this by updating deployment.yaml with the new location.

Add a volume mounts item to the containers array, and set it to /etc/prometheus

Next add a volumes section, which maps from the config map, and match the name of the config map.

The deployments.yaml file, should now look more like this (bolded sections are the bits we just added):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "prometheus.fullname" . }}
  labels:
    {{- include "prometheus.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "prometheus.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "prometheus.selectorLabels" . | nindent 8 }}
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      volumes:
        - name: prometheus-config-volume
          configMap:
            name: prometheus-server-config
      serviceAccountName: {{ include "prometheus.serviceAccountName" . }}
      securityContext:
        {{- toYaml .Values.podSecurityContext | nindent 8 }}
      containers:
        - name: {{ .Chart.Name }}
          volumeMounts:
          - name: prometheus-config-volume
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
          securityContext:
            {{- toYaml .Values.securityContext | nindent 12 }}
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - name: http
              containerPort: 9090
              protocol: TCP
          livenessProbe:
            httpGet:
              path: /
              port: http
          readinessProbe:
            httpGet:
              path: /
              port: http
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

You can double check that this is all in order by executing onto the pod, and looking at the config file. Either use k9s, or run the command:

$ kubectl exec -n monitoring -it prometheus-859f599667-2vvts -- /bin/sh/prometheus $ cd /etc/prometheus//etc/prometheus $ cat prometheus.ymlglobal:
  scrape_interval:     15s
  evaluation_interval: 15sscrape_configs:
  - job_name: prometheus
    metrics_path: /metrics
    scheme: httpstatic_configs:
    - targets: ['localhost:9090']

Updating the Network Policy

Now we have a configured Prometheus instance, we should think about Network Policies.

If you look at the ingress.yaml you’ll notice a lot of templated code, which personally I think we could probably clean up with something a bit easier to follow.

{{- if .Values.ingress.enabled -}}
{{- $fullName := include "prometheus.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
  name: {{ $fullName }}
  labels:
    {{- include "prometheus.labels" . | nindent 4 }}
  {{- with .Values.ingress.annotations }}
  annotations:
    {{- toYaml . | nindent 4 }}
  {{- end }}
spec:
  {{- if .Values.ingress.tls }}
  tls:
    {{- range .Values.ingress.tls }}
    - hosts:
        {{- range .hosts }}
        - {{ . | quote }}
        {{- end }}
      secretName: {{ .secretName }}
    {{- end }}
  {{- end }}
  rules:
    {{- range .Values.ingress.hosts }}
    - host: {{ .host | quote }}
      http:
        paths:
          {{- range .paths }}
          - path: {{ . }}
            backend:
              serviceName: {{ $fullName }}
              servicePort: {{ $svcPort }}
          {{- end }}
    {{- end }}
  {{- end }}

What we want to do, is delete the ingress.yaml and create a file called network.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-egress
spec:
  podSelector:
    matchLabels:
      name: prometheus
  policyTypes:
    - Ingress
    - Egress
  egress:
  - {}
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            project: monitoring

This should allow ingress from the same namespace, which we will need shortly, when we setup an ingress controller, and we allow all traffic egress.

Setting up an ingress controller

The prometheus instance is not very useful if we cannot hit it. There are many different ingress controllers, but for this example we are going to use Trafaek.

Traffic is as described by themselves as, a Cloud-Native Networking Stack That Just Works

To be continued