gRPC Envoy Nghttp2 and Load Balancing

I’ve been using gRPC at work and in several personal projects for months and happy with it, but when it comes to load balancing gRPC does not come with batteries included.

For a long time the only document was the Load Balancing draft in the gRPC repo, the clients should implement a Picker interface to know about the servers, so the pooling and controling the load were handled by the clients.
HTTP/2 was new and most of the reverse proxies implementations were not capable of load balancing gRPC HTTP2 frames, the only solution was to use a TCP load balancer, generating errors, improper and weird behaviours for the clients.

At least two projects are now supporting gRPC load balancing easily.

• The recently announced Envoy from Lyft
• And nghttpx from nghttp2

Here are some notes to simply load balance two gRPC Helloworld server! running on ports 50050 & 50051.

• For nghttp2, a simple configuration file will do

frontend=*,8000;no-tls
backend=localhost,50050;;no-tls;proto=h2;fall=2;rise=2
backend=localhost,50051;;no-tls;proto=h2;fall=2;rise=2
workers=8

• For envoy, here is the cluster part

    "clusters": [
      {
        "name": "local_service",
        "connect_timeout_ms": 250,
        "type": "static",
        "lb_type": "least_request",
        "features": "http2",
        "hosts": [
          {
            "url": "tcp://127.0.0.1:50050"
          },
          {
            "url": "tcp://127.0.0.1:50051"
          }
        ]
      }
    ]

You can then tweak the greeter_client to loop for requests, so you can simulate a client doing multiple requests while killing/restarting your servers.

for {  
    r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: name}) 
    if err != nil { 
        log.Printf("could not greet: %v", err)
    }
    log.Printf("Greeting: %s", r.Message)
}

And modify the greeter_server to show on which port/server you get your response:

// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
    return &pb.HelloReply{Message: "Hello " + in.Name + port}, nil 
}

Those tests aren’t enabling any TLS so use grpc.WithInsecure().

Note that Envoy is also capable of bridging your HTTP/1.1 queries to gRPC, which is a killer feature (I haven’t tested it yet) , you would normally do it by code with gRPC-gateway.

Envoy is really new and I’m still digging into but already proves itself to be a complete load balancing proxy solution with or without gRPC in your stack.