kubernetes nginx ingress with proxy protocol ended up with broken header

Question

I try to setup nginx ingress (nodeport) on google container with proxy protocol so that the real ip can be forwarded to backend service, but ended up with broken header.

2017/02/05 13:48:52 [error] 18#18: *2 broken header: "�����~��]H�k��m[|����I��iv.�{y��Z �嵦v�Ȭq���2Iu4P�z;�    o$�s����"���+�/�,�0̨̩����/" while reading PROXY protocol, client: 10.50.0.1, server: 0.0.0.0:443

If without the proxy protocol, thing works well. According to the https://blog.mythic-beasts.com/2016/05/09/proxy-protocol-nginx-broken-header/ this is due to the protocol v2 is used (binary), but nginx only can speak v1. Any suggestion?

Answer 1

Just ran into this problem myself. For me, I wasn't behind a load balancer (other than my nginx ingress), so I did not actually need proxy-protocol set.

However, I was getting 127.0.0.1 as the client ip still. The trick is that there was a bug in the version of the nginx ingress I was using (0.9.0-beta.5). Updating my container image to gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.8 fixed the issue and I received the proper X-Forwarded-For header.

Note that the higher versions (up to beta.11 at the time of writing this) had the issue remaining, so I've stayed on beta.8 for the time being.

You can see the versions available at https://console.cloud.google.com/gcr/images/google-containers/GLOBAL/nginx-ingress-controller.

If you are wanting to look at the configuration options available, check out https://github.com/kubernetes/ingress/tree/master/controllers/nginx.

Answer 2

I had this problem myself and this was the thing that finally made it work. Updating to version beta.8 of the nginx controller.

In case some people using AWS want to learn from my mistakes, don't go through manual configuration of the load balancer through the aws cli. The above mentioned service annotation does it all for you. I could have saved myself a lot of headache if I had realized that.

Answer 3

GKE: With kubernetes v1.6+ source ip is preserved by default and can be found in headers under x-real-ip without setting any extra nginx config.

AWS: Source ip can be preserved by adding this to the annotations

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
  labels:
    app: nginx-ingress

Checkout this link https://github.com/kubernetes/ingress/tree/master/examples/aws/nginx