Prevent Envoy from modifying the sharding key

Question

We use a two-layer Envoy setup.

[front-end] -> E -> [middleware] -> E -> [backend]

Middleware is supposed to take the sharding key from the HTTP metadata and re-transmit it when talking to the backend.

What we have noticed is that Envoy modifies the HTTP header, which is crashing our service inside gRPC.

E1016 11:19:45.808599731      19 call.cc:912]                validate_metadata: {"created":"@1602847185.808584663","description":"Illegal header value","file":"external/com_github_grpc_grpc/src/core/lib/surface/validate_metadata.cc","file_line":44,"offset":56,"raw_bytes":"36 37 36 38 33 61 34 34 36 35 36 35 37 30 34 33 36 66 36 34 36 35 34 31 34 39 33 61 36 35 36 33 36 63 36 39 37 30 37 33 36 35 32 64 37 30 36 63 37 35 36 37 36 39 36 65 a5 '67683a44656570436f646541493a65636c697073652d706c7567696e.'\u0000"}
E1016 11:19:45.808619606      19 call_op_set.h:947]          assertion failed: false

Any way to avoid this?

UPDATE: Seems to be only happening with x- headers.

Answer 1

The problem was actually not related to Envoy in the end. Turns out that gRPC strings are not null terminated.