Surprising result while transpiling C to Go

H3 is a geospacial indexing library created by Uber. If you are familiar with this blog it’s similar to S2.

Uber is providing h3 for Go. Unfortunately it’s a CGO version, a C to Go binding as you may know CGO is not Go.

Having a native Go library is easier to deal with but would mean a full rewrite.

Transpiling

You may have heard about a recent effort to transpile C to Go using ccgo and the associated libc.

Sqlite has been transpiled using this technique with great success; more recently PCRE2 as well.

The result library is often slower than pure C (~2x magnitude) but still having a native Go port could simplify some aspects of the workflow like testing.

I’m working on a side project where I need to index millions of coordinates using h3 indexation, so I gave ccgo a try, hoping for something working.

Not 100% of the libc is covered, some builtins are not defined I had to define isFinite:

bool isXFinite(double f) { return !isnan(f - f); }

/**
 * Encodes a coordinate on the sphere to the H3 index of the containing cell at
 * the specified resolution.
 *
 * Returns 0 on invalid input.
 *
 * @param g The spherical coordinates to encode.
 * @param res The desired H3 resolution for the encoding.
 * @return The encoded H3Index (or H3_NULL on failure).
 */
H3Index H3_EXPORT(geoToH3)(const GeoCoord* g, int res) {
    if (res < 0 || res > MAX_H3_RES) {
        return H3_NULL;
    }
    if (!isXFinite(g->lat) || !isXFinite(g->lon)) {
        return H3_NULL;
    }

    FaceIJK fijk;
    _geoToFaceIjk(g, res, &fijk);
    return _faceIjkToH3(&fijk, res);
}

The C tests are passing, let’s continue transpiling:

  CC=/usr/bin/gcc ccgo  -pkgname ch3  -trace-translation-units -export-externs X -export-defines D -export-fields F -export-structs S -export-typedefs T -Isrc/h3lib/include  -I../src/h3lib/include ../src/h3lib/lib/*.c

The generated code is visible on my goh3 repo, with some helper functions, it can be used exactly like the original Uber library.

Note that for this experiment I’ve only created the helper functions for the indexation part.

It works!! But no surprise it’s slower, way slower…

1 millions coordinates to cell and back:

• In pure C (-O2): 2.03s
• Using Uber CGO: 2.25s
• Using the transpiled code: 13.07s

Profiling

What is slow? Using the profiler it’s evident the cost to initialize the ccgo libc is too big:

func FromGeo(geo GeoCoord, res int) H3Index {
	tls := libc.NewTLS()
	defer tls.Close()

	cgeo := ch3.TGeoCoord{
		Flat: deg2rad * geo.Latitude,
		Flon: deg2rad * geo.Longitude,
	}

	return H3Index(ch3.XgeoToH3(tls, uintptr(unsafe.Pointer(&cgeo)), int32(res)))
}

What if we could initialize then batch?

func NewBatch() *Batch {
	return &Batch{TLS: libc.NewTLS()}
}

func (c *Batch) FromGeo(geo GeoCoord, res int) H3Index {
	cgeo := ch3.TGeoCoord{
		Flat: deg2rad * geo.Latitude,
		Flon: deg2rad * geo.Longitude,
	}

	return H3Index(ch3.XgeoToH3(c.TLS, uintptr(unsafe.Pointer(&cgeo)), int32(res)))
}

Batching 1 million again: 1.82s

It’s not only working it’s as fast and sometimes faster than the C code, probably due to Go runtime scaling on multiple cores.

I put some benchmarks comparing CGO & CCGO.

cpu: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
BenchmarkToGeoCCGO-8    	32722182	       367.9 ns/op	       0 B/op	       0 allocs/op
BenchmarkToGeoCGO-8     	30941000	       389.8 ns/op	      16 B/op	       1 allocs/op
BenchmarkFromToCCGO-8   	 6580898	      1831 ns/op	       0 B/op	       0 allocs/op
BenchmarkFromToCGO-8    	 5373619	      2230 ns/op	      32 B/op	       2 allocs/op

Interestingly it’s faster on amd64 but a bit slower on arm64 M1.

Conclusion

CCGO is an incredible piece of code with a bright future, it’s one more tool in the Go ecosystem that we can rely on.