Kubernetes: Tomcat throws Exception when enabling proxy protocol

Question

I am kind of lost right now. I set up a Kubernetes Cluster, deployed a Spring Boot API and a LoadBalancer which worked fine. Now I want to enable proxy protocol on the LoadBalancer to preserve the real clients IP, but once I do this my Spring Boot API always returns with a 400 Bad Request and an IllegalArgumentException is thrown.

Here is the short stack trace (I masked the ip addresses):

2020-09-29 20:05:58.382  INFO 1 --- [nio-8080-exec-1] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character found in the HTTP protocol [255.255.255.253 255.255.255.254]
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:560) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.37.jar!/:9.0.37]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

I am using the hcloud-cloud-controller-manager from Hetzner.

Here is my LoadBalancer:

apiVersion: v1
kind: Service
metadata:
  labels:
    service: auth-service
  name: auth-service-service
  annotations:
    load-balancer.hetzner.cloud/name: "lb-backend"
    load-balancer.hetzner.cloud/health-check-port: "80"
    load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
spec:
  ports:
    - name: http
      port: 80
      targetPort: 8080
  selector:
    service: auth-service
  externalTrafficPolicy: Local
  type: LoadBalancer

Here is my Spring Config:

spring:
  datasource:
    platform: postgres
    url: ${DATABASE_CS}
    username: ${DATABASE_USERNAME}
    password: ${DATABASE_PASSWORD}
    driver-class-name: org.postgresql.Driver
  flyway:
    schemas: authservice
  jpa:
    show-sql: false
    properties:
      hibernate:
        dialect: org.hibernate.dialect.PostgreSQLDialect
        jdbc:
          lob:
            non_contextual_creation: true
    hibernate:
      ddl-auto: validate

security:
  jwt:
    secret-key: ${JWT_SECRET_KEY}
    expires: ${JWT_EXPIRES:300000}

mail:
  from: ${MAIL_FROM}
  fromName: ${MAIL_FROM_NAME}
  smtp:
    host: ${SMTP_HOST}
    username: ${SMTP_USERNAME}
    password: ${SMTP_PASSWORD}
    port: ${SMTP_PORT:25}
  mjml:
    app-id: ${MJML_APP_ID}
    app-secret: ${MJML_SECRET_KEY}
stripe:
  keys:
    secret: ${STRIPE_SECRET_KEY}
    public: ${STRIPE_PUBLIC_KEY}
server:
  forward-headers-strategy: native

As you may have noticed I already tried to enable the forward-headers based on this issue.

Thanks for your help!

Answer 1

You enabled the PROXY protocol, which is not HTTP, but a different protocol to tunnel TCP connections to downstream servers, keeping as much information as possible. https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

I am quite sure you want to disable this load-balancer.hetzner.cloud/uses-proxyprotocol: "true" and instead rely on the forward headers to the the remote-address to the correct value for the client.

To be honest I am not aware that tomcat supports the PROXY protocol. (Edit: Currently is does not, see https://bz.apache.org/bugzilla/show_bug.cgi?id=57830)