11 Mar 2019, 00:32

gRPC Load Balancing inside Kubernetes

Context

I wanted to blog about this for years: how to connect to a Kubernete’s loadbalanced service?
How to deal with disconnections/re-connections, maintenance? What about gRPC specifically?
The answer is heavily connected to the network stack used by Kubernetes, but with the “Mesh Network” revolution, It’s not always clear how it works anymore and what the options are.

How it works

First I recommend you to watch this great yet simple video: Container Networking From Scratch, then the Services clusterIP documentation.

To make it simple when you create a Service in Kubernetes, it creates a layer 4 proxy and load balance connections to your pods using iptables, the service endpoint is one IP and a port hiding your real pods.

The Problem

A simple TCP load balancer is good enough for a lot of things especially for HTTP/1.1 since connections are mainly short lived, the clients will try to reconnect often, so it won’t stay connected to an old running pod.

But with gRPC over HTTP/2, a TCP connection is maintained open which could lead to issues, like staying connected to a dying pod or unbalancing the cluster because the clients will end on the older pods.

One solution is to use a more advanced proxy that knows about the higher layers.

Envoy, HAProxy and Traefik are layer 7 reverse proxy load balancers, they know about HTTP/2 (even about gRPC) and can disconnect a backend’s pod without the clients noticing.

Edge

On the edge of your Kubernetes cluster, you need a public IP, provided by your cloud provider via the Ingress directive it will expose your internal service.

To further control your request routing you need an Ingress Controller.
It’s a reverse proxy that knows about the Kubernetes clusters and can direct the requests to the right place. Envoy, HAProxy and Traefik can act as Ingress Controllers.

Internal Services & Service Mesh

In a Micro-services environment, most if not all your micro-services will also be clients to others micro-services.

Istio, a “Mesh Network” solution, use Envoy as a sidecar. This sidecar is configured from a central place (control plane) and makes each micro-service talking to each other through Envoy.

This way the client does not need to know about all the topology.

That’s great but in a controlled environment (yours), where you control all the clients, sending all the traffic through a proxy is not always necessary.

Client Load Balancing

In Kubernetes you can create a headless service; where there are no load balanced single endpoints anymore, the service pods are directly exposed, Kubernetes DNS will return all of them.

Here is an example service called geoipd scaled to 3.

Name:      geoipd
Address 1: 172.17.0.18 172-17-0-18.geoipd.default.svc.cluster.local
Address 2: 172.17.0.21 172-17-0-21.geoipd.default.svc.cluster.local
Address 3: 172.17.0.9 172-17-0-9.geoipd.default.svc.cluster.local

It’s up to your client to connect them all and load balance the connections.

In Go gRPC client side, a simple dns:/// notation will fetch the entries for you, then the roundrobin package will handle load balancing.

conn, err := grpc.Dial(
    "dns:///geoip:9200",
    grpc.WithBalancerName(roundrobin.Name),
)

This may sound like a good solution but it is not: the default refresh frequency is 30 minutes, meaning whenever you add new pods, it can take up to 30 minutes for them to start getting traffic! You can lower this problem by tweaking MaxConnectionAge on the gRPC server:

gsrv := grpc.NewServer(
    // MaxConnectionAge is just to avoid long connection, to facilitate load balancing
    // MaxConnectionAgeGrace will torn them, default to infinity
    grpc.KeepaliveParams(keepalive.ServerParameters{MaxConnectionAge: 2 * time.Minute}),
)

Even if you could refresh the list more often you wouldn’t know about pod eviction fast enough and you’d miss some traffic.

There is a nicer solution, implementing the gRPC client resolver for Kubernetes, talking to the Kubernetes API to get the endpoints and watch them constantly, this is exactly what Kuberesolver does.

// Register kuberesolver to grpc
kuberesolver.RegisterInCluster()

conn, err := grpc.Dial(
    "kubernetes:///geoipd:9200",
    grpc.WithBalancerName(roundrobin.Name),
)

By using kubernetes schema you tell kuberesolver to fetch and watch the endpoints for the geoipd service.

For this to work the pod must have GET and WATCH access to endpoints using a role:

kubectl create role pod-reader-role --verb=get --verb=watch --resource=endpoints,services 
kubectl create sa pod-reader-sa 
kubectl create rolebinding pod-reader-rb --role=pod-reader-role --serviceaccount=default:pod-reader-sa 

Redeploy your app (the client) with the service account:

spec:
  serviceAccountName: pod-reader-sa

Deploy, scale up, scale down, kill your pods, your client is still sending traffic to a living pod !

I’m surprised it’s not mentioned more often, client load balancing did the job for years, the same apply inside Kubernetes environment.
It is fine for small to medium projects and can deal with a lot of traffic, this will do it for many of you unless if you are Netflix sized…

Conclusion

Load-balancing proxies are great tools, especially useful on the edge of your platform. “Mesh Network” solutions are nice additions to our tool set, but the cost of operating and debugging a full mesh network could be really expensive and overkill in some situations, while a client load balancing solution is simple and easy to grasp.

Thanks to Prune who helped me with this post, and to Robteix & diligiant for reviewing.

04 Mar 2019, 00:32

Traefik gRPC Load Balancing and Traces Propagation

Following my recent blog post on setting up a dev environment in Kubernetes, here are some tips to use Traefik as a gRPC load balancer.

Traefik can be used on the edge and route incoming HTTP traffic to your Kubernetes cluster, but it’s also supporting gRPC.



gRPC Load Balancing with Traefik

Here I have a gRPC service I want to expose on the edge.

apiVersion: v1
kind: Service
metadata:
  name: myapp
  labels:
    name: "myapp"
    type: "grpc"
spec:
  ports:
    - port: 9200
      name: "grpc"
      targetPort: grpc
      protocol: TCP
  selector:
    app: "myapp"
  clusterIP: None

Note the clusterIP: None, it’s a headless service.

It will create a non loadbalanced service, pod’s services can be accessed directly.

myapp.default.svc.cluster.local.    2       IN      A       172.17.0.19
myapp.default.svc.cluster.local.    2       IN      A       172.17.0.10
myapp.default.svc.cluster.local.    2       IN      A       172.17.0.16

Here is the ingress for Traefik.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myapp-ingress
  namespace: default
  labels:
    name: "myapp"
  annotations:
    ingress.kubernetes.io/protocol: h2c
spec:
  rules:
  - host: myapp-lb.minikube
    http:
      paths:
      - path: /
        backend:
          serviceName: myapp
          servicePort: 9200

Note the h2c prefix, indicating HTTP2 protocol without TLS to your backend !

Traefik

Tracing

Traefik can be configured to emit tracing.

I’m using ocgrpc Opencensus, for gRPC metrics & traces.
It automatically emits several counters for gRPC and traces using the StatsHandler.

Unfortunately ocgrpc does not yet propagate Jaeger traces, I’ve temporary forked it to support Jaeger.

As you can see you can follow the request from Traefik down to your services.

Jaeger

Happy tracing !

21 Feb 2019, 00:19

Kubernetes Quick Setup with Prometheus, Grafana & Jaeger

Introduction

When starting on a new project or prototyping on a new idea, I find myself doing the same tasks again and again.
Thanks to Kubernetes it’s possible to setup a new env from scratch really fast.

Here is a quick setup (mostly notes) to create a dev environment using Minikube and the workflow I’m using with it.

Not knowing in advance where this future project will be hosted, I try to stay platform agnostic.
OpenCensus or OpenTracing can abstract the target platform, letting you choose what tooling you want for your dev.

I consider some tools to be mandatory these days:

  • Jaeger for tracing
  • Prometheus for instrumentation/metrics
  • Grafana to display these metrics
  • A logging solution: this is already taken care of by Kubernetes and depends on your cloud provider (StackDriver on GCP…), otherwise use another tool like ELK stack.
    On your dev, plain structured logs to stdout with Kubernetes dashboard or Stern should be fine.
  • A messaging system: for example NATS but out of the scope of this post.

Tools Installation

I find it easier to let Minikube open reserved ports, but not mandatory:

minikube start --extra-config=apiserver.service-node-port-range=80-30000

I’ll use Traefik as Ingress to simplify access to several admin UI later.

I’m not a big fan of helm, so here is a little trick, I’m only using helm to create my deployment templates, using the charts repo as a source template, so I can commit or modify the resulting generated files. (Thanks Prune for this tips).

git clone git@github.com:helm/charts.git 

helm template charts/stable/traefik --name traefik --set metrics.prometheus.enabled=true --set rbac.enabled=true \
--set service.nodePorts.http=80 --set dashboard.enabled=true --set dashboard.domain=traefik-ui.minikube > traefik.yaml 

helm template charts/stable/grafana --name grafana --set ingress.enabled=true \ 
--set ingress.hosts\[0\]=grafana.minikube --set persistence.enabled=true --set persistence.size=100Mi > grafana.yaml 

helm template charts/stable/prometheus --name prometheus --set server.ingress.enabled=true \ 
--set server.ingress.hosts\[0\]=prometheus.minikube --set alertmanager.enabled=false \ 
--set kubeStateMetrics.enabled=false --set nodeExporter.enabled=false --set server.persistentVolume.enabled=true \
--set server.persistentVolume.size=1Gi --set pushgateway.enabled=false > prometheus.yaml 

A lot more templates are available: NATS, Postgresql …

For Jaeger a development ready solution exists, here is mine slightly tweaked to use an ingress:

curl -o jaeger.yaml https://gist.githubusercontent.com/akhenakh/615686891340f5306dcbed82dd1d9d67/raw/41049afecafb05bc29de3b0d25208c784f963695/jaeger.yaml

Deploy to Minikube (ensure your current context is minikube…), if you need to work on several projects at the same time remember you can use Kubernetes namespaces, (beware some helm templates are overriding it).

kubectl create -f traefik.yaml
kubectl create -f jaeger.yaml
kubectl create -f grafana.yaml
kubectl create -f prometheus.yaml

Again to ease my workflow I want Traefik to bind 80, edit the service and change it to nodePort 80.

kubectl edit service traefik

Add some urls to your /etc/hosts

echo "$(minikube ip) prometheus.minikube" | sudo tee -a /etc/hosts 
echo "$(minikube ip) grafana.minikube" | sudo tee -a /etc/hosts 
echo "$(minikube ip) traefik-ui.minikube" | sudo tee -a /etc/hosts
echo "$(minikube ip) jaeger.minikube" | sudo tee -a /etc/hosts

Point your browser to any of these addresses and you are good to go !

Deploy your own apps

First remember to always use the Kubernetes Docker:

eval `minikube docker-env` 

My applications are reading parameters from environment, in Go, I’m using namsral/flag, so the flag -httpPort is also set by the environment variable HTTPPORT.

I then use templates, where I set all my environment variables, to create my yaml deployment.

https://gist.github.com/akhenakh/8c06186b7d524f6a60d40764b307a5d5

So typical Makefile targets would fill the template with envsubst:

.EXPORT_ALL_VARIABLES:
VERSION := $(shell git describe --always --tags)
DATE := $(shell date -u +%Y%m%d.%H%M%S)
LDFLAGS := -ldflags "-X=main.version=$(VERSION)-$(DATE)"
CGO_ENABLED=0
GOOS=linux
GOARCH=amd64
PROJET = mysuperproject
...

helloworld: 
	cd helloworld/cmd/helloworld && go build $(LDFLAGS)

helloworld-image: helloworld
	cd helloworld/cmd/helloworld && docker build -t helloworld:$(VERSION) .

helloworld-deploy: NAME=helloworld
helloworld-deploy: helloworld-image 
	cat deployment/project-apps.yaml | envsubst | kubectl apply -f - 
	cat deployment/project-services.yaml | envsubst | kubectl apply -f - 

project-undeploy:
    kubectl delete --ignore-not-found=true deployments,services,replicasets,pods --selector=appgroup=$(PROJECT)

Dockerfile contains nothing but FROM gcr.io/distroless/base and a copy of the helloworld binary.

Note all this setup is only good for your dev, production deployment is another story.

make helloworld-deploy, compilation, image creation and deployment is less than 2s over here!

Shell tool

Another useful trick when working with new tools is to have a shell available inside Kubernetes to experiment with.

Create an image for this purpose, and copy your tools and clients.

FROM alpine:3.9
RUN apk add --no-cache curl busybox-extras tcpdump

WORKDIR /root/
COPY helloworldcli .
ENTRYPOINT ["tail", "-f", "/dev/null"]

And the relevant Makefile targets:

debugtool-image: helloworldcli
	cp helloworld/cmd/helloworldcli/helloworldcli debugtool/helloworldcli
	cd debugtool && docker build  -t debugtool:latest .
	rm -f debugtool/helloworldcli

debugtool-deploy: debugtool-image
	kubectl delete --ignore-not-found=true pod debugtool
	sleep 2
	kubectl run --restart=Never --image=debugtool:latest --image-pull-policy=IfNotPresent debugtool

debugtool-shell:
	kubectl exec -it debugtool -- /bin/sh 

By calling make debugtool-shell, you’ll be connected on a shell inside Kubernetes.

Conclusion

Hope this will help some of you, I would love to hear your own tricks and setup to develop on Minikube !

27 Aug 2018, 00:19

Wasm with Go to build an S2 cover map viewer

I needed a reason to use the new Go 1.11 Wasm port for “real”.

To make it short, it compiles Go code to Wasm binary format for a virtual machine running in web browsers.

I’ve always needed a debug tool to display S2 Cells on a map for different shapes, some online tools already exist:

I’ve planned for a Qt Go app or a QGIS plugin with C++ bindings to Python but to ship those modules would be a nightmare.

I needed another solution, a simple web app would do it, not very complicated but since I hate HTML/CSS/js, I’ve never bothered to start one…
The s2map solution was great but having to start a backend and pay for it, was a no go for the long run, plus since I work with Go I needed something that was relying on the S2 Go port for matching results.

So here I am doing some web dev…

Wasm & Go

First Wasm is not the best solution (GopherJS maybe is) to my problem but hey it’s working.

The main() is a bit weird but close enough to a normal Go program:

func registerCallbacks() {
	js.Global().Set("geocell", js.NewCallback(geoJSONToCells))
}

func main() {
	c := make(chan struct{}, 0)
	println("Wasm ready")
	registerCallbacks()
	<-c
}

Since our function geocell() will be called by js, we wait for a channel that will never be triggered so the main loop won’t return.

NewCallback() wants a fn func(event Value) it means you can’t return directly from Go to js

func geoJSONToCells(i []js.Value)  

Just a slice of untyped values thank you js.

All other functions (not exposed to js) can be normal Go functions, packages …

Interaction with the DOM is very limited via the package syscall/js
So far, to update back the UI (from Go to js), I pass the result of the computation via a set and call the js method, very hackish …

func updateUIWithData(data string) {
	js.Global().Set("data", data)
	js.Global().Call("updateui")
}

The updateui() is a regular js function that processes data and updates the DOM.

App

The app is calling a lot of Go code and libraries that were not written for the web but are now executed from a webpage without any backends:

  • the web interface creates a shape on a js Leaflet layer
  • exports the shape in GeoJSON
  • serializes it to JSON
  • calls the geoJSONToCells() func passing the JSON as string argument
  • computes the S2 cells in the Go world
  • sets the result back via a js var containing GeoJSON as string
  • reads back this GeoJSON and displays it as a Leaflet layer

The first loading and running of the Wasm is very slow on Chrome but not on Firefox, the execution itself is really fast.

ws2

Code is on Github and Demo is hosted on Github Pages at https://s2.inair.space, sorry folks Wasm works on phones but the demo won’t be very useful on mobile.

Size

You can argue 11M (1.5M gzipped) is too big for a webapp providing only one functionality, and you are probably right (then look at a modern webpage), but how big would be a Python app shipped with the C++ library or a full Qt app …

Also the size could be a non-issue, in a near future, a solution like jsgo.io could provide package level CDN caching.

Conclusion

Again you should have really good reasons to use Wasm, the back and forth between js & Wasm is nonsense, but the tooling will improve and I’m sure we will see plenty of solutions around it (one is gRPC in the browser).

EDIT: Call() can call a js method no need for eval !

01 Aug 2018, 00:19

My Own Car System, Rear Camera, Offline Maps & Routing, Map Matching with Go on Raspberry Pi part II

This is my journey building an open source car system with Go & Qt, rear camera, live OpenGL map …

Cross compilation

In Part I, I had to patch qtmultimedia for the camera to work, but Qt compilation is resource hungry, same goes for the osrm compilation, the memory of the Raspberry Pi is too small.

I had to to set up a cross compilation system in my case for armv7h.

QML Development

Since most of the application is in QML, I’ve used the c++ main.cpp launcher as long as possible for the development.
At the moment I needed to inject data from the outside world (like the GPS location) to QML via Qt, so I switched to Go using therecipe Qt Go bindings.

The Go bindings project is young but the main author is really active fixing issues.

It makes desktop applications easy to code without the assle of C++ (at least for me).

About QML, by separating the logic and forms using .qml.ui you still can edit your views with Qt Creator:
That’s just the narrative, truth is Creator is really buggy and I edited the ui files by hand most of the time.
I worked with Interface Builder on iOS for years, Qt is painful, lack of decent visual editor for QML really hurts.

Serving the map without internet access

In Part I, we talked about OpenMapTiles and OpenGL rendering, but I needed a web server capable of reading MBTiles format and serving the necessary assets for the map to be rendered.

I’ve created mbmatch in Go for that purpose so mocs can render the map without Internet access, it will also map match the positions in the future.

Experimenting with another touch screen

I’m using a less performant but smaller LANDZO 5 Inch Touch Display 800x480
This touchscreen is handled as a one button mouse.

It can be calibrate using tslib ts_calibrate command.

Then in your start env tell Qt to use tslib.

TSLIB_TSDEVICE=/dev/input/event0
QT_QPA_GENERIC_PLUGINS=tslib:/dev/input/event0
QT_QPA_FB_TSLIB=1

GPS

Like I said in part I, the Linux gps daemons are using obscure and over complicated protocols so I’ve decided to write my own gps daemon in Go using a gRPC stream interface. You can find it here.

I’m also not satisfied with the map matching of OSRM for real time display, I may rewrite one using mbmatch.

POIs

I’ve started POIs lookups with full text search and geo proximity using bleve by exposing an API compatible with the OSM API so it can be used directly by QML Locations.

Night Map

I’m a huge fan of the Solarized colors, I’ve made a style for the map you can find it here

Solarized

Speeding up boot

systemctl mask systemd-udev-settle.service
systemctl mask lvm2-activation-net.service
systemctl mask lvm2-monitor.service

Status

The project is far from finished and not ready for everybody but it’s fun to play with.

I’ve open sourced most of the code for Mocs on github, feel free to contribute.

10 Jun 2018, 09:19

My Own Car System, Rear Camera, Offline Maps & Routing on Raspberry Pi part I

At first I needed a car rear camera, one thing led to another…

My Car, from 2011, only has an LCD display and no rear camera, so I bought a PAL rear camera, we passed some cables from the rear window to the front then everything begun.
Here is my journey to transform my car into a modern system running on RPi3 (a never ending project).

Hardware

I’m using an Rpi3 (old model).

With Arch for ARM but any system will do.

The screen is an Eleduino Raspberry Pi 7 Inch 1024x600 Pixel IPS Hdmi Input Capacitive Touch Screen Display
An USB 2.0 EasyCap to retrieve the composite signal.

No driver needed for both the screen and the video capture dongle.

mplayer tv:// -tv driver=v4l2:device=/dev/video0:fps=25:outfmt=yuy2:norm=PAL

Camera
mplayer is working out of the box, so I thought everything was okay with the camera, so I thought.

I needed a GUI to display the camera and the date (at this this time this project was just a rear camera display).
So I choose Qt & Golang, not the normal contenders but I can’t handle C++ and had experiences with QtGo, and modern Qt apps are just QML code anyway… So I thought…

I’ve started to code a small QML app but when displaying the video I got:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device '/dev/video0' does not support progressive interlacing
Additional debug info:
gstv4l2object.c(3768): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Device wants interleaved interlacing

Qt via Gstreamer does not allow non interlaced videos :(
One solution is to force a pipeline with an interlacer.

Easy on the command line gst-launch-1.0 v4l2src ! interlace ! xvimagesink

Not that easy via Qt I had to patch the Qt Gstreamer plugin camerabinsession.cpp to insert a filter on the preview: at the end of GstElement *CameraBinSession::buildCameraSource()

    const QByteArray envInterlace = qgetenv("QT_GSTREAMER_CAMERABIN_VIDEO_INTERLACE");
    if (envInterlace.length() > 0) {
        GstElement *interlace = gst_element_factory_make ("interlace", NULL);
        if (interlace == NULL)
            g_error ("Could not create 'interlace' element");


        g_object_set(G_OBJECT(m_camerabin), "viewfinder-filter", interlace, NULL);

        #if CAMERABIN_DEBUG
            qDebug() << "set camera filter" << interlace;
        #endif
        gst_object_unref (interlace);
    }

GPS

I had a serial GPS around, why not display a moving map?

Enable serial port in /boot/config.txt (note Bluetooth must be disabled …)

enable_uart=1
dtoverlay=pi3-disable-bt

pin 8 TXD
pin 10 RXD

remove console=ttyAMA0,115200 and kgdboc=ttyAMA0,115200 from /boot/cmdline.txt

I thought it would be very easy to read NMEA via serial.
It was, gpsd worked in seconds but … it seems we can’t disable the auto speed, equal 4s lost at start.
Plus Qt is using libgeoclue or Gypsy which does not want to talk with gpsd.
I tried both of them, they didn’t work, it’s a mess to debug, documentation is just the API…

So one thing led to another… I’ve written a very small and simple gpsd in Go with a gRPC interface, so it can be queried from anything.
It’s also a bit more advanced since it can map match & route match the positions with OSRM.

Offline maps

OpenMapTiles project is great to generate vectors data in MBTILES format, you can serve them with mbmatch.
Qt Map QML can display them using the mapboxgl driver, some free styles are provided.

Here is an example QML Map plugin.

    Plugin {
        id: mapPlugin
        name: "mapboxgl"
        PluginParameter{
            name: "mapboxgl.mapping.additional_style_urls"
            value: "http://localhost:4000/osm-liberty-gl.style"
        }
    }

Note on X11 and EGL:
Using the mapboxgl renderer under X11 on the Rpi3 is taking a lot of ressources.
Qt5 is capable of directly talking to the GPU without X11, the performance difference is night and day.

So just run your Qt app without X11 with the following env vars.

QT_QPA_PLATFORM=eglfs:/dev/fb0
QT_QPA_EGLFS_HIDECURSOR=yes

Offline Routing

Hopefully the provided Qt osm plugins knows how to route using the OSRM API.
So you can run a local OSRM backend for routing and it will just work.

Generate the route indexes.

osrm-extract -p /usr/share/osrm/profiles/car.lua quebec-latest.osm.pbf 
osrm-contract quebec-latest.osrm
osrm-routed quebec-latest.osrm
    Plugin {
        id: routePlugin
        name: "osm"
        PluginParameter{
            name: "osm.routing.host"
            value: "http://localhost:5000/route/v1/driving/"
        }
    }

Mocs

The app can display the rear camera and a moving map !!

Part 2 will be about searching places by extracting OSM data and indexing them in a small Go program that can run on the rpi, reading ODB data from the car via Bluetooth, packaging the whole thing and open sourcing some code.

23 Apr 2018, 14:37

Bitlbee: Slack, Hangouts & Facebook via IRC gateway

Having multiple clients to handle multiple networks is a mess, especially the Slack client which is really heavy and annoying.

Slack is deprecating its IRC gateway interface on May 15h 2018.

Bitlbee is an IRC server working as a gateway to different IMs.

Optionally Bitlbee can be compiled with LibPurple to support even more networks, like Slack and Hangouts.

Here are some notes for Slack, Facebook and Hangouts to be enabled with Bitlbee.

Slack

You need Slack for libpurple, compiled and installed, it will provide /usr/lib/purple-2/libslack.so.
Restart Bitlbee run, check with help purple you can see * slack (Slack).

account add slack myuser@domain.slack.com
Create a Legacy token.
account slack set api_token xoxp-xxxxxxxxx
account slack on

Join an existing Slack channel:
chat add slack general then join #general

Hangouts, Gtalk

You need Hangouts for libpurple, AUR Arch it will provide /usr/lib/purple-2/libhangouts.so.

account add hangouts youremail@yourdomain.com

Facebook

You need Facebook for Bitlbee.

List existing group chats: fbchats facebook, sometimes it will only display older chats.

Look at the URL in your browser when clicking on a chat on https://messenger.com, grab the id of the group chat and type the command:
chat add facebook 1393844480dd74 #myclub then join #myclub channel.

More Facebook commands.

Apply to all

Autojoining channel on connection:
channel myclub set auto_join true

18 Jan 2018, 08:20

Google S2 with Python & Jupyter

Google is working again on S2 a spatial library !!!

And they even have created a website to communicate about it s2geometry.

The C++ port contains a Python Swig interface.

I’ve been using an unofficial Python port with Jupyter for years now things are way more simpler.

If you are on Arch I’ve create a package, simply install AUR s2geometry-git

First we want a clean Jupyter install from scratch:

virtualenv3 ~/dev/venv3
source ~/dev/venv3/bin/activate
pip install jupyter
pip install cython
pip install numpy
pip install matplotlib scikit-learn scipy Shapely folium geojson Cartopy
cp /usr/lib/python3.6/site-packages/_pywraps2.so $VIRTUAL_ENV/lib/python3.6/site-packages      
cp /usr/lib/python3.6/site-packages/pywraps2.py $VIRTUAL_ENV/lib/python3.6/site-packages

Here is a simple test map.

import folium
import pywraps2 as s2

# create a rect in s2
region_rect = s2.S2LatLngRect(
        s2.S2LatLng.FromDegrees(48.831776, 2.222639),
        s2.S2LatLng.FromDegrees(48.902839, 2.406))

# ask s2 to create a cover of this rect
coverer = s2.S2RegionCoverer()
coverer.set_min_level(10)
coverer.set_max_level(30)
coverer.set_max_cells(60)
covering = coverer.GetCovering(region_rect)
print([c.ToToken() for c in covering])

# create a map
map_osm = folium.Map(location=[48.86, 2.3],zoom_start=12, tiles='Stamen Toner')

# get vertices from rect to draw them on map
rect_vertices = []
for i in [0, 1, 2, 3, 0]:
    vertex = region_rect.GetVertex(i)
    rect_vertices.append([vertex.lat().degrees(), vertex.lng().degrees()])
    
# draw the cells
style_function = lambda x: {'weight': 1, 'fillColor':'#eea500'}
for cellid in covering:
    cell = s2.S2Cell(cellid)
    vertices = []
    for i in range(0, 4):
        vertex = cell.GetVertex(i)
        latlng = s2.S2LatLng(vertex)
        vertices.append([latlng.lng().degrees(),
                         latlng.lat().degrees()])
        
    gj = folium.GeoJson({ "type": "Polygon", "coordinates": [vertices]}, style_function=style_function)
    gj.add_children(folium.Popup(cellid.ToToken()))
    gj.add_to(map_osm)
    
# warning PolyLine is lat,lng based while GeoJSON is not
ls = folium.PolyLine(rect_vertices, color='red', weight=2)
ls.add_children(folium.Popup("shape"))
ls.add_to(map_osm)
    
map_osm

And here is the resulting Jupyter notebook

08 Oct 2017, 14:28

Hacking Temperature Radio Sensors and Graphing with Prometheus

One year ago I’ve started to collect temperature from my house using Acurite sensors.

Acurite outddor Acurite indoor

These sensors are not too expensive and good quality but the “base” aka the radio receiver connected to internet is costly and totally closed, it’s sending your data to the Cloud™, it’s not just Acurite, all those “IOT” devices are generally poor on the software side.

Receiving radio data

Most of these sensors have their protocols already reverse engineered you only need the radio receiver part.

RTL-SDR dongles to the rescue and a great community around it.

Prune998 and I have added some JSON support to the acurite driver and sent it to the projet.

Collecting and graphing the data

The last needed piece was something to collect the data and send to prometheus:

So I wrote a quick Go program to do exactly that Acurite to graph

Et voila, we can graph all sensors in our home.

graph

It has been tested on OSX & Linux.

08 Oct 2017, 09:57

Notes on PacBSD

PacBSD is a FreeBSD kernel/world with a PacMan Arch package manager and an optionnal OpenRC init system.

In short ZFS, DTrace and the FreeBSD kernel but the simplicity of Arch for packages management but the Gentoo init.

It’s experimental, uncompleted, unfinished, lacks proper documentations but it works and could be/should be the solution we are waiting for :).

Here is some notes on installation (in QEMU), note that it slightly diverges from the official install since it’s using a whole ZFS disk, so no GPT.

zpool create tank /dev/vtbd0
zfs create -o canmount=off -o mountpoint=legacy tank/ROOT
zfs create -o canmount=on -o compression=lz4 -o mountpoint=/ tank/ROOT/pacbsd
zfs create -o compression=lz4 -o mountpoint=/home tank/HOME
zfs create -o compression=lz4 -o mountpoint=/root tank/HOME/root
pacstrap /mnt base

Add to loader.conf

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/ROOT/pacbsd"
arch-chroot /mnt
ln -s /usr/share/zoneinfo/zone/subzone /etc/localtime
rc-update add zfs default
echo 'hostname="pacbsd"' > /etc/conf.d/hostname
dd if=/boot/zfsboot of=/dev/vtbd0 count=1
dd if=/boot/zfsboot of=/dev/vtbd0 iseek=1 oseek=1024

Edit loader.conf

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/ROOT/pacbsd"

Reboot, manually starts dhcpd, explore …

dhcpcd vtnet0

Even with you don’t plan on installing PacBSD, the provided ISO is a useful bootable FreeBSD 11 kernel with zfs and pacman/pacstrap tools.