10 Jun 2018, 09:19

My Own Car System, Rear Camera, Offline Maps & Routing on Raspberry Pi part I

At first I needed a car rear camera, one thing led to another…

My Car, from 2011, only has an LCD display and no rear camera, so I bought a PAL rear camera, we passed some cables from the rear window to the front then everything begun.
Here is my journey to transform my car into a modern system running on RPi3 (a never ending project).

Hardware

I’m using an Rpi3 (old model).

With Arch for ARM but any system will do.

The screen is an Eleduino Raspberry Pi 7 Inch 1024x600 Pixel IPS Hdmi Input Capacitive Touch Screen Display
An USB 2.0 EasyCap to retrieve the composite signal.

No driver needed for both the screen and the video capture dongle.

mplayer tv:// -tv driver=v4l2:device=/dev/video0:fps=25:outfmt=yuy2:norm=PAL

Camera
mplayer is working out of the box, so I thought everything was okay with the camera, so I thought.

I needed a GUI to display the camera and the date (at this this time this project was just a rear camera display).
So I choose Qt & Golang, not the normal contenders but I can’t handle C++ and had experiences with QtGo, and modern Qt apps are just QML code anyway… So I thought…

I’ve started to code a small QML app but when displaying the video I got:

ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device '/dev/video0' does not support progressive interlacing
Additional debug info:
gstv4l2object.c(3768): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
Device wants interleaved interlacing

Qt via Gstreamer does not allow non interlaced videos :(
One solution is to force a pipeline with an interlacer.

Easy on the command line gst-launch-1.0 v4l2src ! interlace ! xvimagesink

Not that easy via Qt I had to patch the Qt Gstreamer plugin camerabinsession.cpp to insert a filter on the preview: at the end of GstElement *CameraBinSession::buildCameraSource()

    const QByteArray envInterlace = qgetenv("QT_GSTREAMER_CAMERABIN_VIDEO_INTERLACE");
    if (envInterlace.length() > 0) {
        GstElement *interlace = gst_element_factory_make ("interlace", NULL);
        if (interlace == NULL)
            g_error ("Could not create 'interlace' element");


        g_object_set(G_OBJECT(m_camerabin), "viewfinder-filter", interlace, NULL);

        #if CAMERABIN_DEBUG
            qDebug() << "set camera filter" << interlace;
        #endif
        gst_object_unref (interlace);
    }

GPS

I had a serial GPS around, why not display a moving map?

Enable serial port in /boot/config.txt (note Bluetooth must be disabled …)

enable_uart=1
dtoverlay=pi3-disable-bt

pin 8 TXD
pin 10 RXD

remove console=ttyAMA0,115200 and kgdboc=ttyAMA0,115200 from /boot/cmdline.txt

I thought it would be very easy to read NMEA via serial.
It was, gpsd worked in seconds but … it seems we can’t disable the auto speed, equal 4s lost at start.
Plus Qt is using libgeoclue or Gypsy which does not want to talk with gpsd.
I tried both of them, they didn’t work, it’s a mess to debug, documentation is just the API…

So one thing led to another… I’ve written a very small and simple gpsd in Go with a gRPC interface, so it can be queried from anything.
It’s also a bit more advanced since it can map match & route match the positions with OSRM.

Offline maps

OpenMapTiles project is great to generate vectors data in MBTILES format, you can serve them with mbtileserver.
Qt Map QML can display them using the mapboxgl driver, some free styles are provided.

Here is an example QML Map plugin.

    Plugin {
        id: mapPlugin
        name: "mapboxgl"
        PluginParameter{
            name: "mapboxgl.mapping.additional_style_urls"
            value: "http://localhost:4000/osm-liberty-gl.style"
        }
    }

Note on X11 and EGL:
Using the mapboxgl renderer under X11 on the Rpi3 is taking a lot of ressources.
Qt5 is capable of directly talking to the GPU without X11, the performance difference is night and day.

So just run your Qt app without X11 with the following env vars.

QT_QPA_PLATFORM=eglfs:/dev/fb0
QT_QPA_EGLFS_HIDECURSOR=yes

Offline Routing

Hopefully the provided Qt osm plugins knows how to route using the OSRM API.
So you can run a local OSRM backend for routing and it will just work.

Generate the route indexes.

osrm-extract -p /usr/share/osrm/profiles/car.lua quebec-latest.osm.pbf 
osrm-contract quebec-latest.osrm
osrm-routed quebec-latest.osrm
    Plugin {
        id: routePlugin
        name: "osm"
        PluginParameter{
            name: "osm.routing.host"
            value: "http://localhost:5000/route/v1/driving/"
        }
    }

Mocs

The app can display the rear camera and a moving map !!

Part 2 will be about searching places by extracting OSM data and indexing them in a small Go program that can run on the rpi, reading ODB data from the car via Bluetooth, packaging the whole thing and open sourcing some code.

23 Apr 2018, 14:37

Bitlbee: Slack, Hangouts & Facebook via IRC gateway

Having multiple clients to handle multiple networks is a mess, especially the Slack client which is really heavy and annoying.

Slack is deprecating its IRC gateway interface on May 15h 2018.

Bitlbee is an IRC server working as a gateway to different IMs.

Optionally Bitlbee can be compiled with LibPurple to support even more networks, like Slack and Hangouts.

Here are some notes for Slack, Facebook and Hangouts to be enabled with Bitlbee.

Slack

You need Slack for libpurple, compiled and installed, it will provide /usr/lib/purple-2/libslack.so.
Restart Bitlbee run, check with help purple you can see * slack (Slack).

account add slack myuser@domain.slack.com
Create a Legacy token.
account slack set api_token xoxp-xxxxxxxxx
account slack on

Join an existing Slack channel:
chat add slack general then join #general

Hangouts, Gtalk

You need Hangouts for libpurple, AUR Arch it will provide /usr/lib/purple-2/libhangouts.so.

account add hangouts youremail@yourdomain.com

Facebook

You need Facebook for Bitlbee.

List existing group chats: fbchats facebook, sometimes it will only display older chats.

Look at the URL in your browser when clicking on a chat on https://messenger.com, grab the id of the group chat and type the command:
chat add facebook 1393844480dd74 #myclub then join #myclub channel.

More Facebook commands.

Apply to all

Autojoining channel on connection:
channel myclub set auto_join true

18 Jan 2018, 08:20

Google S2 with Python & Jupyter

Google is working again on S2 a spatial library !!!

And they even have created a website to communicate about it s2geometry.

The C++ port contains a Python Swig interface.

I’ve been using an unofficial Python port with Jupyter for years now things are way more simpler.

If you are on Arch I’ve create a package, simply install AUR s2geometry-git

First we want a clean Jupyter install from scratch:

virtualenv3 ~/dev/venv3
source ~/dev/venv3/bin/activate
pip install jupyter
pip install cython
pip install numpy
pip install matplotlib scikit-learn scipy Shapely folium geojson Cartopy
cp /usr/lib/python3.6/site-packages/_pywraps2.so $VIRTUAL_ENV/lib/python3.6/site-packages      
cp /usr/lib/python3.6/site-packages/pywraps2.py $VIRTUAL_ENV/lib/python3.6/site-packages

Here is a simple test map.

import folium
import pywraps2 as s2

# create a rect in s2
region_rect = s2.S2LatLngRect(
        s2.S2LatLng.FromDegrees(48.831776, 2.222639),
        s2.S2LatLng.FromDegrees(48.902839, 2.406))

# ask s2 to create a cover of this rect
coverer = s2.S2RegionCoverer()
coverer.set_min_level(10)
coverer.set_max_level(30)
coverer.set_max_cells(60)
covering = coverer.GetCovering(region_rect)
print([c.ToToken() for c in covering])

# create a map
map_osm = folium.Map(location=[48.86, 2.3],zoom_start=12, tiles='Stamen Toner')

# get vertices from rect to draw them on map
rect_vertices = []
for i in [0, 1, 2, 3, 0]:
    vertex = region_rect.GetVertex(i)
    rect_vertices.append([vertex.lat().degrees(), vertex.lng().degrees()])
    
# draw the cells
style_function = lambda x: {'weight': 1, 'fillColor':'#eea500'}
for cellid in covering:
    cell = s2.S2Cell(cellid)
    vertices = []
    for i in range(0, 4):
        vertex = cell.GetVertex(i)
        latlng = s2.S2LatLng(vertex)
        vertices.append([latlng.lng().degrees(),
                         latlng.lat().degrees()])
        
    gj = folium.GeoJson({ "type": "Polygon", "coordinates": [vertices]}, style_function=style_function)
    gj.add_children(folium.Popup(cellid.ToToken()))
    gj.add_to(map_osm)
    
# warning PolyLine is lat,lng based while GeoJSON is not
ls = folium.PolyLine(rect_vertices, color='red', weight=2)
ls.add_children(folium.Popup("shape"))
ls.add_to(map_osm)
    
map_osm

And here is the resulting Jupyter notebook

08 Oct 2017, 14:28

Hacking Temperature Radio Sensors and Graphing with Prometheus

One year ago I’ve started to collect temperature from my house using Acurite sensors.

Acurite outddor Acurite indoor

These sensors are not too expensive and good quality but the “base” aka the radio receiver connected to internet is costly and totally closed, it’s sending your data to the Cloud™, it’s not just Acurite, all those “IOT” devices are generally poor on the software side.

Receiving radio data

Most of these sensors have their protocols already reverse engineered you only need the radio receiver part.

RTL-SDR dongles to the rescue and a great community around it.

Prune998 and I have added some JSON support to the acurite driver and sent it to the projet.

Collecting and graphing the data

The last needed piece was something to collect the data and send to prometheus:

So I wrote a quick Go program to do exactly that Acurite to graph

Et voila, we can graph all sensors in our home.

graph

It has been tested on OSX & Linux.

08 Oct 2017, 09:57

Notes on PacBSD

PacBSD is a FreeBSD kernel/world with a PacMan Arch package manager and an optionnal OpenRC init system.

In short ZFS, DTrace and the FreeBSD kernel but the simplicity of Arch for packages management but the Gentoo init.

It’s experimental, uncompleted, unfinished, lacks proper documentations but it works and could be/should be the solution we are waiting for :).

Here is some notes on installation (in QEMU), note that it slightly diverges from the official install since it’s using a whole ZFS disk, so no GPT.

zpool create tank /dev/vtbd0
zfs create -o canmount=off -o mountpoint=legacy tank/ROOT
zfs create -o canmount=on -o compression=lz4 -o mountpoint=/ tank/ROOT/pacbsd
zfs create -o compression=lz4 -o mountpoint=/home tank/HOME
zfs create -o compression=lz4 -o mountpoint=/root tank/HOME/root
pacstrap /mnt base

Add to loader.conf

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/ROOT/pacbsd"
arch-chroot /mnt
ln -s /usr/share/zoneinfo/zone/subzone /etc/localtime
rc-update add zfs default
echo 'hostname="pacbsd"' > /etc/conf.d/hostname
dd if=/boot/zfsboot of=/dev/vtbd0 count=1
dd if=/boot/zfsboot of=/dev/vtbd0 iseek=1 oseek=1024

Edit loader.conf

zfs_load="YES"
vfs.root.mountfrom="zfs:tank/ROOT/pacbsd"

Reboot, manually starts dhcpd, explore …

dhcpcd vtnet0

Even with you don’t plan on installing PacBSD, the provided ISO is a useful bootable FreeBSD 11 kernel with zfs and pacman/pacstrap tools.

22 Jul 2017, 21:28

Arch Linux on a Chromebook Asus C301SA

I’ve got an Asus C301S at work, it’s a Chromebook with Chromeos.

I like those little laptops, for the price it’s actually a very good little machine.
ChromeOS is responding very well, ssh and Chrome are working too.
But I often need more, like X11 forwarding or offline coding …

You will see the C301SA is marked as a C300SA internally.

Here are the steps to install Arch Linux on it, do it at your own risk, you can brick your computer, you will need an external USB keyboard.

  • Activate the recovery mode, by using ESC + F3 (refresh) and the power key

  • Activate the developer mode with ctrl + d, confirm you want to continue, it will take a long time before it’s finished preparing the developer mode.

  • Shutdown and remove the write protection screw
    It’s actually written as not mandatory but also “could be dangerous”, didn’t take the chance.
    Gently dismount the keyboard (by removing the back screws), warning the screws are different, so remember their positions.
    Locate the write protection screw, mine was covered with a black tape, (on the left close to the USB port) and remove it
    c301 screw

  • Reboot into ChromeOS by pressing ctrl + d

  • Patch the firmware [Ctrl+Alt+T] to get a ChromeOS terminal
    type or copy paste cd; curl -LO https://mrchromebox.tech/firmware-util.sh && sudo bash firmware-util.sh
    Choose 1 to enable legacy boot then n to boot from internal then shutdown the computer.

  • Create a bootable USB key for Arch Linux
    sudo dd if=/home/akh/Downloads/archlinux-2017.06.01-x86_64.iso of=/dev/sde bs=512

  • Boot on Arch by pressing ctrl + l then ctrl + u, sometimes it fails

  • The actual Arch kernel does not see the keyboard … you’ll need an external USB keyboard.

  • Perform a normal installation of Arch (I’ve totally removed ChromeOS partitions) and even performed an MBR install (instead of GPT) with only 2 partitions.
    The internal SSD disk is /dev/mmcblk0

  • Install grub
    grub-install --target=i386-pc /dev/mmcblk0
    grub-mkconfig -o /boot/grub/grub.cfg

  • Install yaourt

git clone https://aur.archlinux.org/package-query.git
cd package-query
makepkg -si
cd ..
git clone https://aur.archlinux.org/yaourt.git
cd yaourt
makepkg -si
cd ..
  • Install the galliumos patched kernel, you can go to bed coz it will take a long time to compile
    yaourt --tmp /var/tmp -S aur/linux-galliumos-braswell

The keyboard should work now.
Also install or grab the config file from aur/galliumos-braswell-config for audio support.

If it won’t boot don’t worry you can still boot using the Arch USB key, then remount your system:

mount /dev/mmcblk0p1 /mnt
arch-chroot /mnt

Touch pad config:

Section "InputClass"
    Identifier "Elan Touchpad"
    Driver "libinput"
    MatchIsTouchpad "on"
    Option "Tapping" "on"
    Option "NaturalScrolling" "true"  
    Option "ClickMethod" "clickfinger"
EndSection

Congratulations you have a fully working computer.

EDIT: Arch kernel 4.12 is fully working without the need to use aur/linux-galliumos-braswell, but 4.13 breaks this support, so keep 4.12 for now.

25 Feb 2017, 09:57

From OSX to Linux

I’ve been a long time UNIX user, ditched Microsoft back in the 90s for FreeBSD, Solaris and Linux on the desktop, but when Apple released MacOS X, I’ve used it as a workstation.

For the last years I’ve used Linux desktops but not on my main computer, today here I am switching back to Linux.

This post is not about the reasons I’m switching, they are simple.
My typical work day is mostly about parsing giant files, running VM and Dockers, coding in Go and not about developing for iOS anymore.

I’m using Arch Linux and KDE/Plasma but many items from this list apply to any Linux distributions.

Put your user in the following groups uucp audio input lp.

Bonjour, mDNS and .local

I was used to query the .local domain to ssh my laptop back.

  • Install nss-mdns, add mdns_minimal [NOTFOUND=return] before resolve in /etc/nsswith.conf
  • Install avahi and start avahi-daemon.service.
  • To make your ssh server visible to others, cp /usr/share/doc/avahi/ssh.service /etc/avahi/services/

Try pinging a Mac host on your LAN with ping hostname.local

Google drive

Install kio-gdrive then start Dolphin and go to Network then Google drive and set up your account or by running the shell command: kioclient5 exec gdrive:/.

Emojis in color 💻

Install noto-fonts-emoji and edit .config/fontconfig/fonts.conf as follow

<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
 <alias>
  <family>sans-serif</family>
  <prefer>
   <family>NotoSans</family>
   <family>NotoColorEmoji</family>
   <family>NotoEmoji</family>
  </prefer>
 </alias>
 <alias>
  <family>serif</family>
  <prefer>
   <family>NotoSerif</family>
   <family>NotoColorEmoji</family>
   <family>NotoEmoji</family>
  </prefer>
 </alias>
 <dir>~/.fonts</dir>
</fontconfig>

This setup is working in most applications but can sometimes display weird results in terminals.

2 Factor USB key U2F

I have a cheap Fido U2F key but Chrome was unable to see it. Edit /etc/udev/rules.d/50-fido-u2f.rules

# this udev file should be used with udev 188 and newer
ACTION!="add|change", GOTO="u2f_end"
# Key-ID FIDO U2F
KERNEL=="hidraw*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="096e", ATTRS{idProduct}=="0850|0880", TAG+="uaccess"
LABEL="u2f_end"

Win key aka ⌘ cmd key

On a PC keyboard the left alt and the windows key are inverted in opposite to a Mac where alt is left of ⌘.
To avoid being lost when I switch back to my Macbook I’ve physically inverted the keys and change the behavior in Plasma in “Hardware | Input Devices”.

⌘

Compose key

To type special or accentuated characters you’ll use the “Compose” key.
You can set the compose key in Plasma in “Hardware | Input Devices”.

Compose

I’m using the right alt, which is the right ⌘ cmd on a Mac (and also a compose key).

For a complete list of compose shortcuts see the bottom of this page

Spotlight

The tool/window equivalent to running Spotlight is called “Plasma Search”, you can configure what it can search for.
The mapping for this key in “System Settings | Global Shortcuts | Run Command”.

System Settings | Global Shortcuts | Run Command

It’s also capable of indexing files content, in KDE/Plasma this service is provided by baloo, ignored directories can be set by calling “Configure File Search”.

You can empty your baloo index by stopping by killing all your baloo processes and rm -r .local/share/baloo then restart for indexation balooctl start

Exposé and active corners

It’s called “Screen Edges” and it’s under “Windows behavior”.

edges

Alternative for Dash

Dash was part of my workflow to get documentation, an alternative solution is to install zeal it’s using the exact same docsets as Dash.
Also see DevDocs

Samba share

Install samba, tweak /etc/samba/smb.conf and enable nmbd.service.
I personally prefer a different password than my shell account: smpasswd -a yourusername.

Taking Screenshot

Install spectacle and use the Ptr Sc key (this shortcut can be setup in Plasma).

Webcam, Hangout and Skype

I have a Logitech C920, it worked without any configuration, inside Chrome so Hangout and even with Skype for Linux.

Mounting a macOS disk

That one is weird …
mount -o ro,sizelimit=498876809216 /dev/sda2 /mnt/OSX

If your existing partition was big, you need to find this magic number by following this guide

Steam

Steam needs x86 32 bits libraries, on Arch you have to enable multilib by editing /etc/pacman.conf

[multilib]
Include = /etc/pacman.d/mirrorlist

Then install the steam package.

Virtualbox / QEMU

I prefer QEMU/KVM & libvirt-manager, when available, over Virtualbox, it’s more integrated into the system, and it’s capable of emulating other cpu architectures like aarch64…

One more huge advantage for QEMU it’s also capable of booting a virtual macOS X

Printing

Install cups and print-manager then enable org.cups.cupsd.service, note that you need the .local resolution above for network printer resolution.
Also install hplib for HP printers.

Share your session aka remote desktop

Install x11vnc and run x11vnc -usepw -once -noxdamage -ncache 10 from your X session.

Note that this vnc server is not compatible with macOS X embedded vnc viewer (vnc://hostname), here is one for Chrome RealVNC

Remember vnc protocol is not secure and you must use an SSH tunnel over it.

Magic Trackpad

I’m using a magic trackpad 2, after enabling bluetooth and pairing the trackpad, one, two & three fingers touches, vertical & horizontal scroll worked via hid_magicmouse module.

For gestures you need libinput-gestures, here is an example /etc/libinput-gestures.conf file:

gesture swipe left 3 xdotool key control+Right

gesture swipe right 3 xdotool key control+Left

gesture swipe up 3 xdotool key control+F9

The bad

  • Not as nice, not as well integrated, for example supporting HiDPI with only one retina screen is weird with Xorg.
  • Key shorcuts are a giant mess under Linux, every applications have their own and it can’t be configured centrally.
  • I’m still missing some applications like Sketch, but most of all Tower when dealing with a git merge issue.

The good

After 8 years of absence on Linux as main desktop, things have changed, it’s not free from bugs but way more simpler to use now than before, and way more configurable than macOS X.

It won’t work for everybody but I’m really happy with this setup, the gain compared to a Mac is big, first the machine itself a 4Ghz i7 with 64G of ram does not even exist at Apple (Hackintosh is not a good solution), ZFS, native Docker over ZFS, better OpenGL (faster fps in games), well maintained packages over Brew/Macports, well maintained drivers, my work is easier …

Notes

20 Oct 2016, 08:58

Telegraf & Prometheus Swiss Army Knife for Metrics

There are a lot of different solutions when it comes to collecting metrics, I found myself happy with this hybrid solution.

Telegraf is an agent written in Go for collecting metrics from the system it’s running on.
It’s developed by Influxdata the people behind InfluxDB, but Telegraf has a lot of outputs plugins and can be used without InfluxDB.
Many different platform (FreeBSD, Linux, x86, Arm …) are offered and only one single static binary (Thanks to Golang) is needed to deploy an agent.

Prometheus is a time series database for your metrics, with an efficient storage.
It’s easy to deploy, no external dependencies, it’s gaining traction in the community because it’s a complete solution, for example capable of discovering your targets inside a Kubernete cluster.

Here is a simple configuration to discover both products.

You can deploy the agent on every hosts you want to monitor but need only one Prometheus running.

Install Prometheus

Download Prometheus for your platform and edit a config file named prometheus.yml.

scrape_configs:
  - job_name: 'telegraf'
    scrape_interval: 10s
    static_configs:
      - targets: ['mynode:9126']

Prometheus is a special beast in the monitoring world, the agents are not connecting to the server, it’s the opposite the server is scrapping the agents.
In this config we are creating a job called telegraf to be scrapped every 10s connecting to mynode host on port 9126.

That’s all you need to run a Prometheus server, start it by specifying a path to store the metrics and the path of the config file:

prometheus -storage.local.path /opt/local/var/prometheus -config.file prometheus.yml

The server will listen on port 9090 for the HTTP console.

Install Telegraf

Download Telegraf agent for your platform and edit telegraf.conf.

[[outputs.prometheus_client]]
    listen = "127.0.0.1:9126"

# Read metrics about cpu usage
[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics.
  collect_cpu_time = false

# Read metrics about memory usage
[[inputs.mem]]

# Read metrics about network interface usage
[[inputs.net]]
  ## By default, telegraf gathers stats from any up interface (excluding loopback)
  ## Setting interfaces will tell it to gather these explicit interfaces,
  ## regardless of status.
  interfaces = ["en2"]

Remember this is the node agent “client” but since Prometheus server will connect it, you are providing a listening endpoint.
Starts the agent with telegraf -config telegraf.conf

There are many more inputs plugins for telegraf for example you can monitor all your Docker instance.

# Read metrics about docker containers
[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  ## Only collect metrics for these containers, collect all if empty
  container_names = []
  ## Timeout for docker list, info, and stats commands
  timeout = "5s"

  ## Whether to report for each container per-device blkio (8:0, 8:1...) and
  ## network (eth0, eth1, ...) stats or not
  perdevice = false
  ## Whether to report for each container total blkio and network stats or not
  total = false

It’s also capable of monitoring third parties product like MySQL, Cassandra

Read Metrics

Prometheus is provided with a visual HTTP console & query tool available on port 9090.

The query language is described here.

The console can’t be really used as a dashboard, you can use Grafana which can speak directly to prometheus.

Install or run Grafana with docker

docker run -i -p 3000:3000 -e "GF_SECURITY_ADMIN_PASSWORD=mypassword"  grafana/grafana

Point your browser to the port 3000.
Add a Prometheus data source and point the host to your Prometheus server port 9090.

Then create your dashboard, here are some queries to display the telegraf agents:

  • CPU
    cpu_usage_idle{host="myhost", cpu="cpu-total"}
    cpu_usage_user{host="myhost", cpu="cpu-total"}
    cpu_usage_system{host="myhost", cpu="cpu-total"}

  • Memory
    mem_available_percent{host="myhost"}

  • Docker Memory
    docker_container_mem_usage_percent
    Legend format {{container_name}}

  • Docker CPU
    docker_container_cpu_usage_percent
    Legend format {{container_name}}

Graph prometheus

Conclusion

You can easily instrument your own development using the client libraries.
There is also a Prometheus gateway for the short lived jobs, so you can batch to the gateway between the scrap period.

It’s a simple setup but capable of handling a lot of data in different contexts, system monitoring & instrumentation.

28 Sep 2016, 13:15

gRPC wrong types context and Go 1.7

If you are following Go development you probably know that:
Go 1.7 moves the golang.org/x/net/context package into the standard library as context, Yeah !
Unfortunately it won’t work for everything, I’ve spent some time understanding this one.

For example if you are using gRPC you can hit this problem, here is an interface generated by gRPC:

type APRSServer interface {
    GetPastMessages(context.Context, *Point) (*ARPSMessages, error)
}

But when compiling:

/main.go:153: cannot use &s (type *Server) as type protorpc.APRSServer in argument to protorpc.RegisterAPRSServer:                                                                           
        *Server does not implement protorpc.APRSServer (wrong type for GetPastMessages method)                                                                                                
                have GetPastMessages("context".Context, *protorpc.Point) (*protorpc.ARPSMessages, error)                                                                                      
                want GetPastMessages("golang.org/x/net/context".Context, *protorpc.Point) (*protorpc.ARPSMessages, error)   

The compiler is complaining about wrong types for the context argument.
Problem is gRPC generated code is importing context as golang.org/x/net/context, that’s the only way it remains compatible between Go 1.6 & Go 1.7.

So a quick solution is to import context in your own code to use the old path:

// we have to use the old context path here for gRPC compat see https://github.com/grpc/grpc-go/issues/711
context "golang.org/x/net/context"

Also note that go tool fix is now capable of fixing the import path with -force context.

22 Sep 2016, 15:39

Using Go mobile on iOS for real

Go Mobile can generate native framework for iOS and Android using Go code, I was curious what could be achieved with it.
Most tutorials are Hello world and I wanted to test it with real code.
You can use it to generate a full app only using Go code, but I’m only interested by the bindings part (SDK applications), using a native ObjC/Swift app calling Go code.

I’m using some existing Go code regionagogo, (a geofence database), moderately complex since it uses BoltDB and Google S2 library.

Go Mobile is limited to a subset of types you can use, main reason is to be correctly transcribed to Java and ObjC.

So the first thing to do is to write some Go wrapper like you would do in ObjC to call some C++ code but first:

Functions must return either no results, one result, or two results where the type of the second is the built-in error type.
So far no major complication, but also note that you can’t return slices, that could be a major pain, there are some workaround solutions like go-mobile-collection that can generate an API to operate on slice.
You also have to respect some naming for your constructor like New...().

At the end your wrapper won’t look very Goish but it’s what it takes for it to be translated.

Knowing those constraints you can write a simple wrapper like this:

package mobile

type GeoDB struct {
	db regionagogo.GeoFenceDB
}

func NewGeoDB() *GeoDB {
	g := &GeoDB{}
	return g
}

func (g *GeoDB) OpenDB(path string) error {
	db, err := rbolt.NewGeoFenceBoltDB(path)
	if err != nil {
		return err
	}
	g.db = db
	return nil
}

...

Then use the gomobile command:

gomobile bind -target=ios github.com/akhenakh/regionagogo/mobile

It generates a native iOS Mobile.Framework, drop it into your XCode project and start using it.

  • GeoDB translates to GoMobileNewGeoDB() and returns a GoMobileGeoDB* in ObjC domain.
  • OpenDB(path string) error to - (BOOL)openDB:(NSString*)path error:(NSError**)error.

A simple example would be:

GoMobileGeoDB *db = GoMobileNewGeoDB();
NSString *resourcePath = [[NSBundle mainBundle] pathForResource:@"region" ofType:@"db"];
        
NSError *error;
[db openDB:resourcePath error:&error];
if (error != nil) {
    NSLog(@"error opening db %@", [error localizedDescription]);
    return;
}

It’s performing very well, on my iPhone 6, around 4,000 queries per second to test a position in a small fence, to 60,000 queries in a hit miss (calling Go overhead is not that big), while using a ridiculously small amount of memory.

I’ve made a demo iOS app where you can hit the map and it tells you in which fence you are.

app
Remember this is running locally on your phone without any network access (but the map), the geo computation and the Polygons are returned by Go code.

Until now I had to maintain libraries in both languages, Go mobile is a nice alternative!

Sources for the wrapper are available in regionagogo gomobile branch, and here is the iOS demo app.