Setting up a Prometheus monitoring stack in Kubernetes, utilising helm.

Pre-requisites
In this article, I will be working with the following software, it makes sense to have these pre-installed before continuing.
- minikube - minikube is local Kubernetes, focusing on making it easy to learn and develop for Kubernetes
- kubectl - The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters.
- k9s - K9s provides a terminal UI to interact with your Kubernetes clusters.
- helm - helm Charts help you define, install, and upgrade even the most complex Kubernetes configurations
You can install all by using brew if on OSX, or check out the website for detailed installation instructions.
Creating a Namespace
Kubernetes supports multiple virtual clusters backed by the same physical cluster. These virtual clusters are called namespaces.
Having a namespace is going to make your life much simpler, when it comes to adding applications to be monitored, as they will be segregated.
So to do this, first of all, create a file called namespace.yaml
Whether you keep this separate or with your application is up to you. Both use cases are common practice, I prefer to keep all namespace definitions separate to application stacks, so when you see examples below, it will be separated out.
Here's how I intend to layout my code:
.
├── application
├── cluster
│ └── namespace.yaml
└── monitoring
Cluster level config is stored under cluster
Monitoring level config under monitoring
Applications config under application
Running in changes
In order to create the namespace (and all other configs going forward), we need to use the kubectl command.
Since the structure is segregated into directories, we need to cd to the cluster directory, and run the following command:
$ cd cluster$ kubectl apply -f *.yaml
namespace/monitoring created$ kubectl get namespaces
NAME STATUS AGE
default Active 355d
kube-node-lease Active 355d
kube-public Active 355d
kube-system Active 355d
monitoring Active 65s
You should now have a running namespace
Creating a Helm Template
We are going to use helm to help us with the prometheus stack, it will save us a huge amount of time in the long run, as it will template the app for us, and make it more manageable.
Run the following command:
$ helm create prometheus
This will create a new helm chart, and will have a structure similar to this:
.
├── Chart.yaml
├── charts
├── templates
│ ├── NOTES.txt
│ ├── _helpers.tpl
│ ├── deployment.yaml
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── service.yaml
│ ├── serviceaccount.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml
Chart.yaml and values.yaml will define what the chart is, and what values will be in it at deployment.
From here, we can deploy so you can test the help chart works. By default it will run nginx, and you should be able to hit one deployed. (I have shown in debug mode so you can see all the resources being created)
$ helm upgrade \
prometheus . \
-n monitoring \
--install \
--atomic \
--debug="true" \
--dry-run="false"history.go:53: [debug] getting history for release prometheus
Release "prometheus" does not exist. Installing it now.
install.go:172: [debug] Original chart version: ""
install.go:189: [debug] CHART PATH: /Users/craiggoddenpayne/Dropbox/Work/BeardyDigital/code/k8s-metrics-stack/prometheusclient.go:109: [debug] creating 4 resource(s)
wait.go:53: [debug] beginning wait for 4 resources with timeout of 5m0s
wait.go:225: [debug] Deployment is not ready: monitoring/prometheus. 0 out of 1 expected pods are ready
NAME: prometheus
LAST DEPLOYED: Sat Oct 31 12:37:45 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
USER-SUPPLIED VALUES:
{}COMPUTED VALUES:
affinity: {}
autoscaling:
enabled: true
maxReplicas: 3
minReplicas: 1
targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
repository: nginx
imagePullSecrets: []
ingress:
annotations: {}
enabled: false
hosts:
- host: chart-example.local
paths: []
tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 1
resources: {}
securityContext: null
service:
port: 80
type: ClusterIP
serviceAccount:
annotations: {}
create: true
name: ""
tolerations: []HOOKS:
---
# Source: prometheus/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "prometheus-test-connection"
labels:
helm.sh/chart: prometheus-0.1.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['prometheus:80']
restartPolicy: Never
MANIFEST:
---
# Source: prometheus/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-0.1.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
---
# Source: prometheus/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-0.1.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
---
# Source: prometheus/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-0.1.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
selector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
template:
metadata:
labels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
spec:
serviceAccountName: prometheus
securityContext:
{}
containers:
- name: prometheus
securityContext:
null
image: "nginx:1.16.0"
imagePullPolicy:
ports:
- name: http
containerPort: 80
protocol: TCP
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
resources:
{}
---
# Source: prometheus/templates/hpa.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-0.1.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "1.16.0"
app.kubernetes.io/managed-by: Helm
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: prometheus
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace monitoring $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace monitoring port-forward $POD_NAME 8080:$CONTAINER_PORT
To hit the pod, you will need to expose it to the host, and to do this you will need to get the pod name by running:
$ kubectl get pod -n monitoringNAME READY STATUS RESTARTS AGE
prometheus-54d964dbb7-qhztr 1/1 Running 0 3m33s
Then use the following command to port forward to the host:
kubectl port-forward -n monitoring pods/prometheus-54d964dbb7-qhztr 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
Handling connection for 8080
So now, container port 80, is mapped to port 8080 on your local machine.
Now, hit that in a browser

Updating the Helm Chart to setup Prometheus
If we have a nosy around in the values.yaml file, you will see where nginx is referenced.
We should be able to change this, to point to a Prometheus image.
Update this in the values.yaml file
image:
repository: docker.io/prom/prometheus
And make sure to update the application version in Chart.yaml
appVersion: v2.20.0
Next update the service port from 80, to 9090
service:
type: ClusterIP
port: 9090
And make sure to updates the probes in deployment.yaml to look at port 9090, or the container will keep cycling
ports:
- name: http
containerPort: 9090
protocol: TCP
Run the deploy again to make sure that there are no errors:
$ helm upgrade \
prometheus . \
-n monitoring \
--install \
--atomic \
--debug="true" \
--dry-run="false"history.go:53: [debug] getting history for release prometheus
Release "prometheus" does not exist. Installing it now.
install.go:172: [debug] Original chart version: ""
install.go:189: [debug] CHART PATH: /Users/craiggoddenpayne/Dropbox/Work/BeardyDigital/code/k8s-metrics-stack/prometheusclient.go:109: [debug] creating 4 resource(s)
wait.go:53: [debug] beginning wait for 4 resources with timeout of 5m0s
wait.go:225: [debug] Deployment is not ready: monitoring/prometheus. 0 out of 1 expected pods are ready
NAME: prometheus
LAST DEPLOYED: Sat Oct 31 14:14:20 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
USER-SUPPLIED VALUES:
{}COMPUTED VALUES:
affinity: {}
autoscaling:
enabled: true
maxReplicas: 3
minReplicas: 1
targetCPUUtilizationPercentage: 80
fullnameOverride: ""
image:
repository: docker.io/prom/prometheus
imagePullSecrets: []
ingress:
annotations: {}
enabled: false
hosts:
- host: chart-example.local
paths: []
tls: []
nameOverride: ""
nodeSelector: {}
podAnnotations: {}
podSecurityContext: {}
replicaCount: 1
resources: {}
securityContext: null
service:
port: 9090
type: ClusterIP
serviceAccount:
annotations: {}
create: true
name: ""
tolerations: []HOOKS:
---
# Source: prometheus/templates/tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "prometheus-test-connection"
labels:
helm.sh/chart: prometheus-1.0.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "v2.20.0"
app.kubernetes.io/managed-by: Helm
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['prometheus:9090']
restartPolicy: Never
MANIFEST:
---
# Source: prometheus/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-1.0.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "v2.20.0"
app.kubernetes.io/managed-by: Helm
---
# Source: prometheus/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-1.0.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "v2.20.0"
app.kubernetes.io/managed-by: Helm
spec:
type: ClusterIP
ports:
- port: 9090
targetPort: http
protocol: TCP
name: http
selector:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
---
# Source: prometheus/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-1.0.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "v2.20.0"
app.kubernetes.io/managed-by: Helm
spec:
selector:
matchLabels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
template:
metadata:
labels:
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
spec:
serviceAccountName: prometheus
securityContext:
{}
containers:
- name: prometheus
securityContext:
null
image: "docker.io/prom/prometheus:v2.20.0"
imagePullPolicy:
ports:
- name: http
containerPort: 9090
protocol: TCP
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
resources:
{}
---
# Source: prometheus/templates/hpa.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: prometheus
labels:
helm.sh/chart: prometheus-1.0.0
app.kubernetes.io/name: prometheus
app.kubernetes.io/instance: prometheus
app.kubernetes.io/version: "v2.20.0"
app.kubernetes.io/managed-by: Helm
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: prometheus
minReplicas: 1
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
Awesome, and lets run a port forward to check that it’s actually working:
$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-8644c59db5-w6c8k 1/1 Running 0 106s$ kubectl port-forward -n monitoring pods/prometheus-8644c59db5-w6c8k 8080:9090Forwarding from 127.0.0.1:8080 -> 9090
Forwarding from [::1]:8080 -> 9090

Cool so now we have a running prometheus pod, defined within a helm chart.
Reconfiguring Prometheus
So having a default prometheus instance is great, but we need to add customisations to make it useful.
The first thing we should look at, is the configuration file.
At the moment it is using the default value supplied from within the docker image.
Lets swap this out with a config map.
Create a file called prometheus-config-map.yaml
Add the following contents:
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-server-config
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: prometheus
metrics_path: /metrics
scheme: http
static_configs:
- targets: ['localhost:9090']
Now we need to tell the docker container to swap out the volume and file, with the above config map, so the container sees our config rather than the default config.
You can do this by updating deployment.yaml with the new location.
Add a volume mounts item to the containers array, and set it to /etc/prometheus
Next add a volumes section, which maps from the config map, and match the name of the config map.
The deployments.yaml file, should now look more like this (bolded sections are the bits we just added):
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "prometheus.fullname" . }}
labels:
{{- include "prometheus.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "prometheus.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "prometheus.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
- name: prometheus-config-volume
configMap:
name: prometheus-server-config
serviceAccountName: {{ include "prometheus.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/prometheus.yml
subPath: prometheus.yml
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 9090
protocol: TCP
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
You can double check that this is all in order by executing onto the pod, and looking at the config file. Either use k9s, or run the command:
$ kubectl exec -n monitoring -it prometheus-859f599667-2vvts -- /bin/sh/prometheus $ cd /etc/prometheus//etc/prometheus $ cat prometheus.ymlglobal:
scrape_interval: 15s
evaluation_interval: 15sscrape_configs:
- job_name: prometheus
metrics_path: /metrics
scheme: httpstatic_configs:
- targets: ['localhost:9090']
Updating the Network Policy
Now we have a configured Prometheus instance, we should think about Network Policies.
If you look at the ingress.yaml you’ll notice a lot of templated code, which personally I think we could probably clean up with something a bit easier to follow.
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "prometheus.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include "prometheus.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ . }}
backend:
serviceName: {{ $fullName }}
servicePort: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
What we want to do, is delete the ingress.yaml and create a file called network.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all-egress
spec:
podSelector:
matchLabels:
name: prometheus
policyTypes:
- Ingress
- Egress
egress:
- {}
ingress:
- from:
- namespaceSelector:
matchLabels:
project: monitoring
This should allow ingress from the same namespace, which we will need shortly, when we setup an ingress controller, and we allow all traffic egress.
Setting up an ingress controller
The prometheus instance is not very useful if we cannot hit it. There are many different ingress controllers, but for this example we are going to use Trafaek.
Traffic is as described by themselves as, a Cloud-Native Networking Stack That Just Works
To be continued