Google endpoints is ignoring headers with underscore key

Question

I have integrated my k8s deployment with google endpoints. I have 12 services out of which 11 services are working fine with google endpoints except for one service. When its transferring requests to application container its ignoring the header values with underscore.

Implementation I have added this in my application deployment (Ignore syntax i have it in correct format in deployment)

- args:
  - --http_port
  - "8081"
  - --backend
  - 127.0.0.1:8080
  - --service
  - <MY_ENDPOINT>
  - --service_account_key
  - /etc/nginx/creds/<MY_CREDS>
  - --rollout_strategy=managed
  image: gcr.io/endpoints-release/endpoints-runtime:1
  imagePullPolicy: Always
  name: esp
  ports:
  - containerPort: 8081
    protocol: TCP
  resources: {}
  volumeMounts:
  - mountPath: /etc/nginx/creds
    name: service-account-creds
    readOnly: true

Here in the below sample, i have sent two headers with TEST-CODE and TEST_CODE. Both are getting received at the application end when i am sending request via application port, but when i access it via esp port, i receive only header with -(hyphen) not with _(underscore). TEST-CODE is being received in the server but not TEST_CODE, when accessing via esp.

When i am sending request to my application using curl or postman via 8080 which is application port

curl -XPOST http://localhost:8080/<CONTEXT_PATH> -H "Content-Type: application/json" -H "USER-ID: abc" -H "TEST-CODE: xyz" -H "TEST_CODE: wxyz"

I get response like this

{
    "timestamp": "2020-04-05T19:42:44.958564Z",
    "principal": null,
    "session": null,
    "request": {
        "method": "POST",
        "uri": "http://localhost:8080/<MY_URL>",
        "headers": {
            "host": [
                "localhost:8080"
            ],
            "test-code": [
                "xyz"
            ],
            "content-type": [
                "application/json"
            ],
            "x-user-id": [
                "abc"
            ],
            "test_code": [
                "wxyz"
            ],
            "user-agent": [
                "curl/7.52.1"
            ],
            "accept": [
                "*/*"
            ]
        },
        "remoteAddress": null
    },
    "response": {
        "status": 200,
        "headers": {
            "Content-Length": [
                "435"
            ],
            "Date": [
                "Sun, 05 Apr 2020 19:42:45 GMT"
            ],
            "Content-Type": [
                "application/json;charset=UTF-8"
            ]
        }
    },
    "timeTaken": 633
}

When i am sending request to my application using curl or postman via 8081 which is via esp

curl -XPOST http://localhost:8081/<CONTEXT_PATH>?key=<MY_KEY> -H "Content-Type: application/json" -H "USER-ID: abc" -H "TEST-CODE: xyz" -H "TEST_CODE: wxyz"

I get response like this

{
    "timestamp": "2020-04-05T19:42:57.102184Z",
    "principal": null,
    "session": null,
    "request": {
        "method": "POST",
        "uri": "http://localhost/<MY_URL>?<MY_KEY>",
        "headers": {
            "x-real-ip": [
                "127.0.0.1"
            ],
            "x-google-real-ip": [
                "127.0.0.1"
            ],
            "x-cloud-trace-context": [
                "<MY_GOOGLE_TRACE_CONTEXT>"
            ],
            "host": [
                "localhost"
            ],
            "test-code": [
                "xyz"
            ],
            "x-endpoint-api-project-id": [
                "<MY_ENDPOINT_PROJECT_ID>"
            ],
            "content-type": [
                "application/json"
            ],
            "x-forwarded-for": [
                "127.0.0.1"
            ],
            "x-user-id": [
                "abc"
            ],
            "user-agent": [
                "curl/7.52.1"
            ],
            "accept": [
                "*/*"
            ]
        },
        "remoteAddress": null
    },
    "response": {
        "status": 400,
        "headers": {
            "Connection": [
                "close"
            ],
            "Content-Length": [
                "256"
            ],
            "Date": [
                "Sun, 05 Apr 2020 19:42:57 GMT"
            ],
            "Content-Type": [
                "application/json"
            ]
        }
    },
    "timeTaken": 21
}

Again,

Notice the difference : I am sending test code header as TEST-CODE and TEST_CODE, both are coming when i am accessing via application port, but when i am accessing via esp container, the application is receiving TEST-CODE alone, not TEST_CODE.

I have tried some other HEADERS with -(hyphen) and some with _(underscore). Every time headers with -(hyphen) is being received via esp to application but not with _(underscore).

I have took the above responses from spring actuator endpoints (actuator/httptrace)

Can someone help?