Envoy Gateway In Production

Feb 05, 2024 - kubernetes envoy proxy gateway

Previous post was about discovering a new offer in the Kubernetes Gateway space Envoy Gateway, in this one I’ll share some notes to make it to production.

Envoy Gateway is still rough on the edges, but remember Gateway is mainly a Kubernetes API frontend to provision Envoy Proxy, Envoy Proxy configuration can still be patched to enable a specific feature.

Enable Patching

Patching is not enabled by default, create this configmap if you need it:

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true

Compression

This very basic functionality is not enabled by default with 0.6.0 Gateway.

This is a patch for an HTTPS listener, the path to patch is different for HTTP "/default_filter_chain/filters/0/typed_config/http_filters/0")

Example patch to enable compression:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: compressor-patch-policy
  namespace: envoy-gateway-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      name: envoy-gateway-system/eg/https
      operation:
        op: add
        path: "/filter_chains/0/filters/0/typed_config/http_filters/0"
        value:
          name: "envoy.filters.http.compressor"
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor"
            compressor_library:
              name: text_optimized
              typed_config:
                "@type": "type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip"
                compression_level: BEST_SPEED
                compression_strategy: DEFAULT_STRATEGY
                memory_level: 8
                window_bits: 15
                chunk_size: 4096
            response_direction_config:
              remove_accept_encoding_header: true
              common_config:
                enabled:
                  default_value: true
                  runtime_key: response_direction_config_enabled
                content_type:
                  - "application/javascript"
                  - "application/json"
                  - "application/x-web-app-manifest+json"
                  - "application/xhtml+xml"
                  - "application/xml"
                  - "font/opentype"
                  - "image/svg+xml"
                  - "image/x-icon"
                  - "text/css"
                  - "text/html"
                  - "text/plain"
                  - "text/xml"
                min_content_length: 60
            request_direction_config:
              common_config:
                enabled:
                  default_value: false
                  runtime_key: request_direction_config_enabled

You should see a response with an content-encoding: gzip header for an HTTP/1.1 request with a Accept-Encoding: deflate, gzip header.

It should be easier in future releases with this PR.

Logging

To enable JSON logging:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  telemetry:
    accessLog:
      settings:
        - format:
            type: JSON
            json:
              authority: "%REQ(:AUTHORITY)%"
              bytes_received: "%BYTES_RECEIVED%"
              bytes_sent: "%BYTES_SENT%"
              duration: "%DURATION%"
              method: "%REQ(:METHOD)%"
              path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
              status: "%RESPONSE_CODE%"
              upstream_host: "%UPSTREAM_HOST%"
              user_agent: "%REQ(USER-AGENT)%"
              x_forwarded_for: "%REQ(X-FORWARDED-FOR)%"
              x_request_id: "%REQ(X-REQUEST-ID)%"
              cluster: "%UPSTREAM_CLUSTER%"
          sinks:
            - type: File
              file:
                path: /dev/stdout

Note the cluster field, it will really help debugging your installation, it will generate the path to the actual container: "cluster":"httproute/mycontainer/mycontainer-backend-open-0/rule/0"

CORS

Enabling CORS is well documented but one detail was missing, if for some reasons the routes are matching on methods (GET,PUT …), Envoy Proy won’t be able to respond to preflights HTTP requests, since they are using OPTIONS:

The documentation has been updated to reflect that: The targeted HTTPRoute or the HTTPRoutes that the targeted Gateway routes to must allow the OPTIONS method for the CORS filter to work. Otherwise, the OPTIONS request won’t match the routes and the CORS filter won’t be invoked.

Regex Match

By default, Envoy Proxy is using a very small limit for the regex complexity, it means a very simple regex may be rejected.
Envoy Gateway (as of 0.6.0) does not increase that value, you should probably do it by changing the default bootstrap config.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 1
  bootstrap:
    type: Merge
    value: |
      layered_runtime:
        layers:
        - name: "static-runtime"
          static_layer:
            re2.max_program_size.error_level: 1000

The error message would be something like this:

[2024-01-30 17:38:22.851][1][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected: regex '^\/v1\/myapi\/([0-9a-z-]{36})\/blah$' RE2 program size of 120 > max program size of 100 set for the error level threshold. Increase configured max program size if necessary.

It could be a huge issue, if for some reasons Envoy is restarting, it will respond with 404 for all the routes until the route is removed or the threshold is increased.

Tracking the issue.

External Authorization (Authz)

I’m running a 0.6.0 fork with gRPC authz enabled.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: myapp-back-policy
  namespace: myapp
spec:
  extAuthz:
    grpcURI: http://ext-auth.envoy-gateway-system.svc.cluster.local:10003
  jwt:
    providers:
    - audiences:
      - https://my.domain.net
      issuer: https://id.issuer-domain.net
      name: auth0
      remoteJWKS:
        uri: https://myid.issuer-domain.net/.well-known/jwks.json
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: myapp-back-authz-0

Note that in 0.6.0, after the JWT filter the payload was not forwarded anymore, it has been changed since (my fork includes that change).

An Extauthz solution by the Gateway team is in development and should land soon.

OIDC OpenID

Not in 0.6.0 but already merged, OIDC is supported to protect an HTTPRoute via a SecurityPolicy.

With Azure (but similar with other providers), register an application, get the client ID (application ID) and the client secret, the issuer URL is built with the UUID from “Directory (tenant) ID”. Set the Redirect URL to this form: https://myapp.mydomain.com/oauth2/callback

On Envoy Gateway side, apply the SecurityPolicy:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: myapp-oidc-policy
  namespace: myapp
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: myapp-public-route
  oidc:
    provider:
      issuer: "https://login.microsoftonline.com/4410211c-15ba-427f-bdf7-cd04eb1aa75c/v2.0"
    clientID: "4e550576-cd82-4566-a00c-4c1fee89f808"
    clientSecret:
      name: "myapp-oidc-client-secret"

Envoy Gateway will try to discover most of the configuration (Authorization & Token Endpoints) from the provider’s Well-Known Configuration Endpoint.

Conclusion

Most if not any of those required extra configurations will probably disappear with the 1.0 release.
Gateway is flexible enough to provide those already.

For now using the main trunk is surely not for everyone, but 1.0.0 will definetly be prod ready.