02 Aug 2015, 16:26

Offering free internet well almost and survive disaster, power outage, Internet blackout

In my neighborhood, I’m experimenting to give free access to some services, not the full internet but a full access to Wikipedia, maps…
Here are some tips do to the same.

Getting access to the internet may be crucial for our lives, but the commercial providers are here to make money out of it. They don’t event provide disasters nor minimum safety access.

Sometimes even in normal condition, while traveling your roaming will cost you hundreds just to get access to a map or Wikipedia.

With a simple RaspberryPi, a wifi antenna and a good geographical position you can reach hundreds of people.

Wifi

To serve wifi to others you need to get a wifi USB dongle that can be set in AP mode.
I’m using an Alfa network one.

To enable the AP mode you need to install the package hostapd. Here is my /etc/hostapd/hostapd.conf.

ssid=freewifi
interface=wlan0
auth_algs=3
channel=7
hw_mode=b
logger_stdout=-1
logger_stdout_level=2
country_code=CA

(Always ensure you are using the correct country to avoid disturbing others and look for a free channel.)

You also need a DNS & DCHP provider, install dnsmasq, my /etc/dnsmasq.conf.

no-resolv
no-poll
server=/localnet/10.4.0.1
address=/#/10.4.0.1
interface=wlan0
bind-interfaces
dhcp-range=10.4.0.10,10.4.0.200,12h

It means all DNS queries will be intercepted and will respond with the same IP 10.4.0.1, by extend every HTTP request will be redirected to 10.4.0.1.

Enable the same IP on the vlan0 interface.
ifconfig wlan0 10.4.0.1

You are all set, no need for NAT no need for routing, we just want to provide access to Wikipedia.

Content

I’m using Gozim to run a full offline copy of Wikipedia. (For example, make it listen on port 8080)

Install nginx to reverse proxy to your content to a fake domain call wikipedia.wifi (port 8080) or serve any directories you want to be published.

Wispr

Another issue while running your own wifi AP is that these days almost all traffic is secured by HTTPS so there is no way to intercept these calls with the right certificates, the users will just get a security error page.
To solve that you need to display a popup page like the one you get when you connect to a Starbucks free wifi network.

I’m working on a set of tool to do just that, uninstall nginx and give it a shot the project is called WisprGo and the code is on Github too.
It’s a work in progress any help is appreciated.

02 Aug 2015, 13:10

Access OS metrics from Golang

I’ve recently published StatGo, it gives you access to your operating system metrics like free memory, used disk spaces …

It’s a binding to the C library libstatgrab, a proven stable piece of code that works on many different systems, FreeBSD, Linux, OSX …

It’s very simple to use:

s := NewStat()
c := s.CPUStats()
fmt.Prinln(c.Idle)
98.2

Feel free to contribute, it may need some improvement but it’s working I’m using it in a small metrics web server to monitor small network of servers.

The code is on Github

02 Aug 2015, 00:32

Host your blog on Github with autodeploy

I’ve always developed my own blog system, that’s a good way to learn a new langage.
But having to maintain a working server or hosting is no fun, there are some solutions like Jekyll or Hugo they generate static web pages based on some Markdown files you wrote.

As it’s just basic html files, they can be served by Github gh-pages.
It opens the door to blogging from anywhere without internet connection or your own laptop, just write some Markdown then publish to github later or event edit your new blog post from the Github editor.
Coupled with a Wercker auto deploy, publishing is automated, no excuse anymore.

Here is some tips for this to work smoothly with Hugo.

Github setup

First create 2 repositories on Github, one for the HTML pages, one for the markdown itself.
Let’s call them blog & hugo-blog.

DNS setup

You can use your own domain, if so you need to enable a CNAME file for your gh-pages to the blog repo, then add a CNAME to your DNS provider:
blog.mydomain.com. IN CNAME username.github.io.

Hugo setup

Clone the hugo-blog repo, put your new hugo blog files in it (created via hugo new).
Choose a theme for your blog (don’t forger to remove the .git directory from it) and set it up in your config.yml as follow:

baseurl = "http://blog.mydomain.com/"
languageCode = "en-us"
title = "My supa blog"
canonifyurls = true
theme = "hyde"

Then run hugo server --buildDrafts, point your browser to http://localhost:1313, no need to reload all modifications appear directly.

Wercker setup

For the auto deploy to occur after each commits you need a build & deploy system, Wercker is very cool as you can reproduce the exact same system on your host with Docker, subscribe to the service (it’s free) and register your hugo-blog repo, hit next, next …

Add a wercker.yml file in your hugo-blog that looks exactly like this.

box: debian
build:
  steps:
    - arjen/hugo-build:
        version: "0.14"
        theme: purehugo
        flags: --disableSitemap=true
deploy :
  steps :
    - script:
        name: Configure git
        code: |-
          sudo apt-get -y update
          sudo apt-get -y install git-core
          git config --global user.email "pleasemailus@wercker.com"
          git config --global user.name "wercker"

          # remove current .git folder
          rm -rf .git
    - script:
        name: Deploy to Github pages
        code: |-
          cd public
          # if you are using a custom domain set it here
          echo "blog.mydomain.com" > CNAME
          git init
          git add .
          git commit -m "deploy commit from $WERCKER_STARTED_BY"
          git push -f $GIT_REMOTE master:gh-pages 2> /dev/null

It will use Hugo to generate the pages then deploy to your Github repo on every commit to the blog repo.

Last step is to get a token from Github for the deploy to occur without your credentials.
Go to your Github settings, Personal access tokens, generate a token.

Go to your Wercker app settings, Deploy targets, add a new target, check auto deploy successful builds to branch(es): and type master.
Add a new variable named GIT_REMOTE check protected and type https://{TOKEN}@github.com/yourusername/blog.git replace {TOKEN} with the token from Github.
You can use this outdated blogpost from Wercker for the screenshot but don’t follow what they said.

You are all set, happy blogging, you can check the exact same config on my repos under my username Github@akhenakh.

01 Aug 2015, 22:36

Migrate to Hugo

This blog is running Hugo with an auto deploy via Wercker and hosted on Github Page. Take #2

Previous blog was hosted on Google App Engine with a Python blog system, to get the previous articles, I had to run in a small data migration.

Make a backup from GAE admin web interface: Go to Datastore admin and backup your entity mine was Post and then backup to blobstore, go to Blob Viewer and download your file named around datastore_backup_datastore_backup_2015_08_02_Post-157413521680733022360302ADC43E4-output-1-attempt-1

I put the migration code I’ve used here, it’s super ugly but helped me to migrate from GAE to Hugo, so it may be help you too.

I’ve used html2text to convert my HTML data back to Markdown.

import sys
import os
import json
import html2text
import errno
import datetime

sys.path.append('/usr/local/google_appengine')
from google.appengine.api.files import records
from google.appengine.datastore import entity_pb
from google.appengine.api import datastore

def mkdir_p(path):
    try:
        os.makedirs(path)
    except OSError as exc:
        if exc.errno == errno.EEXIST and os.path.isdir(path):
            pass
        else: raise

raw = open("datastore", 'r')
titles = open("titles.txt", 'r').readlines()
reader = records.RecordsReader(raw)
i = 0
for record in reader:
    entity_proto = entity_pb.EntityProto(contents=record)
    entity = datastore.Entity.FromPb(entity_proto)

    if entity.get("status") == 1:
        path = titles[i].rstrip()
        content = html2text.html2text(entity["content_html"], "http://blog.nobugware.com/")
        directory = os.path.dirname(path)
        mkdir_p("post" + directory)

        # get current local time and utc time
        localnow = datetime.datetime.now()
        utcnow = datetime.datetime.utcnow()

        # compute the time difference in seconds
        tzd = localnow - utcnow
        secs = tzd.days * 24 * 3600 + tzd.seconds

        # get a positive or negative prefix
        prefix = '+'
        if secs < 0:
            prefix = '-'
            secs = abs(secs)

        # print the local time with the difference, correctly formatted
        suffix = "%s%02d:%02d" % (prefix, secs/3600, secs/60%60)
        now = localnow.replace(microsecond=0)
        date = "%s%s" % (entity["creation_date"].isoformat(' '), suffix)
        tags_string = ""
        tags_cleaned = []
        if entity.get("tags") is not None:
            tags = entity.get("tags")
            for tag in tags:
                tags_cleaned.append("\""+ tag + "\"")
            tags_string = ",".join(tags_cleaned)

        print tags_string
        page = """+++
date = "%s"
title = "%s"
tags = [%s]
+++

%s
""" % ( date, entity["title"] , tags_string, content)
    md=open("post" + path + ".md", 'w')
    md.write(page.encode('utf8'))
    md.close()
    i = i + 1

02 Apr 2015, 12:49

A 10 minutes walk into Grafana & Influxdb

This is a 10 minute tutorial to set up an InfluxDB + Grafana with Go on your Mac, but should work with minor modifcations on your favorite Unix too, it assumes you already have a working Go compiler.

InfluxDB is a database specialized into time series, think store everything associated with a time, makes it perfect for monitoring and graphing values. Grafana is a js frontend capable of reading the data from InfluxDB and graphing it.

brew install influxdb

Start InfluxDB, and then point your browser to http://localhost:8083 default user is root, password is root and default port is 8086.

influxdb -config /usr/local/etc/influxdb.conf Create a database called test.

Let’s test the connection with the db and Go, first install the InfluxDB driver for Go:

go get github.com/influxdb/influxdb/client Test your setup with some code:

package main

import (
    "fmt"

    "github.com/influxdb/influxdb/client"
)

func main() {
    c, err := client.NewClient(&client.ClientConfig{
        Username: "root",
        Password: "root",
        Database: "test",
    })

    if err != nil {
        panic(err)
    }

    dbs, err := c.GetDatabaseList()
    if err != nil {
        panic(err)
    }

    fmt.Println(dbs)
}

If you are good you should see a map containing all your created InfluxDB databases.

Now let’s measure something real: the time it takes for your http handler to answer.

package main

import (
    "fmt"
    "log"
    "math/rand"
    "net/http"
    "time"

    "github.com/influxdb/influxdb/client"
)

var c *client.Client

func mySuperFastHandler(rw http.ResponseWriter, r *http.Request) {
    start := time.Now()
    // sleeping some random time
    rand.Seed(time.Now().Unix())
    i := rand.Intn(1000)
    time.Sleep(time.Duration(time.Duration(i) * time.Millisecond))
    fmt.Fprintf(rw, "Waiting %dms", i)
    t := time.Since(start)

    // sending the serie
    s := &client.Series{
        Name:    "myhostname.nethttp.mySuperFastHandler.resp_time",
        Columns: []string{"duration", "code", "url", "method"},
        Points: [][]interface{}{
            []interface{}{int64(t / time.Millisecond), 200, r.RequestURI, r.Method},
        },
    }
    err := c.WriteSeries([]*client.Series{s})
    if err != nil {
        log.Println(err)
    }
}

func main() {
    var err error
    c, err = client.NewClient(&client.ClientConfig{
        Username: "root",
        Password: "root",
        Database: "test",
    })
    if err != nil {
        panic(err)
    }

    http.HandleFunc("/", mySuperFastHandler)
    http.ListenAndServe(":8080", nil)
}

This is not very useful as it’s measuring the time to write to the ResponseWriter that’s why I’ve added some random time but you get the sense. It will save a serie per request as: duration, status code, url, http method, the name of the serie is important as many tools (as Graphite) are using the dots as separator, so think twice before naming your serie. Point your browser to http://localhost:8080 and reload the page several times.

Now that we have data let’s browse them with the InfluxDB browser, go to the InfluxDB admin and hit “explore data” and select with:

SELECT duration FROM myhostname.nethttp.mySuperFastHandler.resp_time WHERE code = 200;

Image Alt

You should be able to see the inserted data points.

Now let’s work with Grafana, download the tar gz, uncompress it somewhere, copy this demo config.js file in the root directory of Grafana. Go to the InfluxDB admin with your browser and add a new database called “grafana”.

In your web browser, open the file index.html in the Grafana directory, you should see a the Grafana interface edit the default graph, enter the query as follow:

  • click on series it will complete with myhostname.nethttp.mySuperFastHandler.resp_time
  • In alias type $0 $2, it will use the 1st part and the 3rd part of the name (remember the dots) so it will display myhostname mySuperFastHandler
  • Finally click on mean and choose duration in the completion, then add code = 200 as where clause.

Hit save and you are done !

Image Alt

There is so much more you can do with InfluxDB & Grafana, it’s really simple to collect and display, hope you want to go further after this. You can look at my generic net/http handler for InfluxDB on Github that can be integrated into your code.

02 Jan 2015, 17:50

Run a full Wikipedia copy from any computers

Today I’m releasing Gozim a side project written in Go.

It’s a set of tools to serve ZIM files, (compressed copy of Wiki articles), use it to run your full copy of Wikipedia off the grid.

It runs great on a small computer as the Raspberry Pi or your own laptop.

It could be a solution to give access to knowledge in countries without stable internet connections.

13 Dec 2014, 07:55

Airspy on Linux

Airspy is an SDR with amazing specs, but drivers are slowly coming to your prefered os.

This apply to Arch but should apply to any recent Linux.

I’ve first compiled libairspy but always had the error AIRSPY_ERROR_NOT_FOUND and "usbfs: interface 0 claimed by airspy while 'airspy_info' sets config #1"

Since Linux 3.17 comes with an airspy v4l autoloaded driver making impossible to use it with libusb: no airspy_info and no gqrx.

Simply get rid of it with a modprobe config like /etc/modprobe.d/airspy.conf

blacklist airspy

You need some deps: the airspy lib, gnuradio-osmosdr (gr-osmosdr) and a recent gqrx

Here are my Arch AUR pkg that will build airspy-git, gr-osmosdr-git and gqrx-git (note that you need to uninstall the packages from community to avoid conflict).

Have fun

28 Nov 2014, 20:29

FreeBSD on Google Compute Engine

First you need to create a VirtualBox FreeBSD install using a 10G qcow format, use an SCSI controller for the install as the disk will be visible as da0 inside GCE.

On FreeBSD 10.1 I had to load virtio manually, so set this in /boot/loader.conf

`virtio_load="YES"
virtio_pci_load="YES"
virtio_blk_load="YES"
if_vtnet_load="YES"`

Copy your ssh key in your home user .ssh/authorized_keys, be sure to be in the wheel group.

On a Mac you need to install GNU tar (brew install gnu-tar), shutdown your VirtualBox vm and upload your image to GCE

`VBoxManage clonehd -format RAW ../VirtualBox\ VMs/FreeBSDGCE/BSDGCE.qcow
~/Downloads/disk.raw
gtar -Szcf freebsd.tar.gz disk.raw
gsutil mb gs://bsdimage
gsutil cp freebsd.tar.gz gs://bsdimage/gce-bsd.tar.gz
gcutil addimage freebsd  gs://bsdimage/gce-bsd.tar.gz`

You should now see “freebsd” as available install image in your console

28 Jul 2014, 17:36

Simple APRS KK6NXK

APRS is a tactical digital communications system used between amateurs radio, to exchange positions & messages, here I blog my experience decoding/encoding APRS with a small Arduino as it may help some of you too.

Some transceivers are incorporating this functionalities but most of them don’t, a lot of new technicians start with cheap Baofeng radios (30$) which don’t provide advanced functionnalities but here is a way to solve that.

I’ve first looked at Bertos project, a realtime os for micro controllers as Atmel 328p (Arduinos), you can see my 1st attempt there:

This was the receiver only, a simple divider, the “hard” part was to understand the Baofeng/Kenwood mic jacks.

It takes me a long time to figure out that the Baofeng was too slow to trigger the squelch, so the Arduino missed the beginning of the message, hopefully I have a Kenwood too …

So for a Baofeng using this kind of APRS decoder, you have to disable the squelsh completly, and set the volume very high it has to reach a proper signal on the arduino around -3v/+3v.

For the AFSK modulation, the Bertos project use a DAC made with 4 resistors.

Later, I’ve discovered the wonderful work of Mark Qvist MicroAPRS, mostly the same circuit as the Bertos project but with an awesome documentation of the code, it’s a pleasure to read & learn.

So I’ve started to work on a circuit using a Bluno Nano an Arduino Nano with Bluetooth Low Energy, as an iPhone developer, this piece of electronics is just incredible and gives me the capabilites to connect my circuits to iphone Apps.

It’s working but hardly a mobile solution :)

Later I’ve designed a PCB for it you can find the full Gerber files in my Github.

I’m using Frizting to design and then export to OSHPark for the PCB manufacturing, it’s far from perfect but I will improve the circuit and publish it soon !

This plus an iPhone app and you’ll rock APRS !

73 de KK6NXK

08 Jul 2014, 01:22

Got my amateur radio license KK6NXK

When I was a young child, I remember listening to an old radio receiver with my grandfather, looking for morse transmission.

30 years later, I take the opportunity to get my radio amateur license in the US, it was not difficult (Technicial license is easy), especially cause US radio amateurs are using the metric system (almost all the time …), the only difficulties were to learn some terms and simple formulas from french to english and to know some parts of regulation by heart (I have never been able to learn by heart).

A lot of people keep asking me why are you doing this? Do you want to speak to a microphone to a stranger?

First why not, it’s not the craziest thing I’ve ever done :)

Then they probably have forgotten that every single piece of technology today are using radio waves, from your 4G phone to wifi to Bluetooth, learning and understanding that is really useful.

For example Radio amateur bands overlap some of the “civil” bands, it gives some extended privileges to a radio amateur to experiment with long distance wifi transmissions.

So it’s not just about voice but data too.

Another aspect is electronic, a lot of radio amateur are sharing hardware and electronic circuits to build your own receiver/transceiver, before it was cool on the Internet.

Learning electronic to build something real is a lot more rewarding than it was at school…

Even without a license this is a brand new world to explore, receiving satellite images of the Earth is one of the many possibilities you can get entering radio world.

This is the beginning of a new knowledge branch to conquer !

See my work on an APRS bluetooth Modem

73 de KK6NXK

By the way in french it’s radio amateur and not amateur radio :)