Kubernetes Quick Setup with Prometheus, Grafana & Jaeger

- Kubernetes devops Go

Introduction

When starting on a new project or prototyping on a new idea, I find myself doing the same tasks again and again.
Thanks to Kubernetes it’s possible to setup a new env from scratch really fast.

Here is a quick setup (mostly notes) to create a dev environment using Minikube and the workflow I’m using with it.

Not knowing in advance where this future project will be hosted, I try to stay platform agnostic.
OpenCensus or OpenTracing can abstract the target platform, letting you choose what tooling you want for your dev.

I consider some tools to be mandatory these days:

Tools Installation

I find it easier to let Minikube open reserved ports, but not mandatory:

minikube start --extra-config=apiserver.service-node-port-range=80-30000

I’ll use Traefik as Ingress to simplify access to several admin UI later.

I’m not a big fan of helm, so here is a little trick, I’m only using helm to create my deployment templates, using the charts repo as a source template, so I can commit or modify the resulting generated files. (Thanks Prune for this tips).

git clone git@github.com:helm/charts.git 

helm template charts/stable/traefik --name traefik --set metrics.prometheus.enabled=true --set rbac.enabled=true \
--set service.nodePorts.http=80 --set dashboard.enabled=true --set dashboard.domain=traefik-ui.minikube > traefik.yaml 

helm template charts/stable/grafana --name grafana --set ingress.enabled=true \ 
--set ingress.hosts\[0\]=grafana.minikube --set persistence.enabled=true --set persistence.size=100Mi > grafana.yaml 

helm template charts/stable/prometheus --name prometheus --set server.ingress.enabled=true \ 
--set server.ingress.hosts\[0\]=prometheus.minikube --set alertmanager.enabled=false \ 
--set kubeStateMetrics.enabled=false --set nodeExporter.enabled=false --set server.persistentVolume.enabled=true \
--set server.persistentVolume.size=1Gi --set pushgateway.enabled=false > prometheus.yaml

A lot more templates are available: NATS, Postgresql …

For Jaeger a development ready solution exists, here is mine slightly tweaked to use an ingress:

curl -o jaeger.yaml https://gist.githubusercontent.com/akhenakh/615686891340f5306dcbed82dd1d9d67/raw/41049afecafb05bc29de3b0d25208c784f963695/jaeger.yaml

Deploy to Minikube (ensure your current context is minikube…), if you need to work on several projects at the same time remember you can use Kubernetes namespaces, (beware some helm templates are overriding it).

kubectl create -f traefik.yaml
kubectl create -f jaeger.yaml
kubectl create -f grafana.yaml
kubectl create -f prometheus.yaml

Again to ease my workflow I want Traefik to bind 80, edit the service and change it to nodePort 80.

kubectl edit service traefik

Add some urls to your /etc/hosts

echo "$(minikube ip) prometheus.minikube" | sudo tee -a /etc/hosts 
echo "$(minikube ip) grafana.minikube" | sudo tee -a /etc/hosts 
echo "$(minikube ip) traefik-ui.minikube" | sudo tee -a /etc/hosts
echo "$(minikube ip) jaeger.minikube" | sudo tee -a /etc/hosts

Point your browser to any of these addresses and you are good to go !

Deploy your own apps

First remember to always use the Kubernetes Docker:

eval `minikube docker-env`

My applications are reading parameters from environment, in Go, I’m using namsral/flag, so the flag -httpPort is also set by the environment variable HTTPPORT.

I then use templates, where I set all my environment variables, to create my yaml deployment.

https://gist.github.com/akhenakh/8c06186b7d524f6a60d40764b307a5d5

So typical Makefile targets would fill the template with envsubst:

.EXPORT_ALL_VARIABLES:
VERSION := $(shell git describe --always --tags)
DATE := $(shell date -u +%Y%m%d.%H%M%S)
LDFLAGS := -ldflags "-X=main.version=$(VERSION)-$(DATE)"
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
PROJET = mysuperproject
...

helloworld: 
	cd helloworld/cmd/helloworld && go build $(LDFLAGS)

helloworld-image: helloworld
	cd helloworld/cmd/helloworld && docker build -t helloworld:$(VERSION) .

helloworld-deploy: NAME=helloworld
helloworld-deploy: helloworld-image 
	cat deployment/project-apps.yaml | envsubst | kubectl apply -f - 
	cat deployment/project-services.yaml | envsubst | kubectl apply -f - 

project-undeploy:
    kubectl delete --ignore-not-found=true deployments,services,replicasets,pods --selector=appgroup=$(PROJECT)

Dockerfile contains nothing but FROM gcr.io/distroless/base and a copy of the helloworld binary.

Note all this setup is only good for your dev, production deployment is another story.

make helloworld-deploy, compilation, image creation and deployment is less than 2s over here!

Shell tool

Another useful trick when working with new tools is to have a shell available inside Kubernetes to experiment with.

Create an image for this purpose, and copy your tools and clients.

FROM alpine:3.9
RUN apk add --no-cache curl busybox-extras tcpdump

WORKDIR /root/
COPY helloworldcli .
ENTRYPOINT ["tail", "-f", "/dev/null"]

And the relevant Makefile targets:

debugtool-image: helloworldcli
	cp helloworld/cmd/helloworldcli/helloworldcli debugtool/helloworldcli
	cd debugtool && docker build  -t debugtool:latest .
	rm -f debugtool/helloworldcli

debugtool-deploy: debugtool-image
	kubectl delete --ignore-not-found=true pod debugtool
	sleep 2
	kubectl run --restart=Never --image=debugtool:latest --image-pull-policy=IfNotPresent debugtool

debugtool-shell:
	kubectl exec -it debugtool -- /bin/sh

By calling make debugtool-shell, you’ll be connected on a shell inside Kubernetes.

Conclusion

Hope this will help some of you, I would love to hear your own tricks and setup to develop on Minikube !