Celery flower with apache2 reverse proxy ssl handshake fails for websocket

Question

I am trying to setup a reverse proxy for flower behind apache2 as outlined in this post Celery Flower Security in Production.

Https pageloads from flower work just fine through the proxy, but the websocket it opens to update the dashboard does not - the ssl handshake fails and no response is returned. I am using apache2 2.2.22 on ubuntu 12.04 and I have added mod_proxy_wstunnel via a commonly used patch in order to tunnel the websocket connection. I am running flower through supervisord on 127.0.0.1:5555.

Here is the relevant portion of my apache .conf

SSLProxyEngine On
SSLProxyVerify none

ProxyRequests Off
ProxyPass /update-dashboard wss://127.0.0.1:5555/update-dashboard
ProxyPassReverse /update-dashboard wss://127.0.0.1:5555/update-dashboard

ProxyPass / http://127.0.0.1:5555/
ProxyPassReverse / http://127.0.0.1:5555/
<Proxy  *>
Order deny,allow
Allow from all
</Proxy>

Here is the request after it eventually times out

"GET /update-dashboard HTTP/1.1"
 500 419 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML,     like Gecko) Chrome/43.0.2357.124 Safari/537.36"

Here is the output from the apache2 error log

[Fri Jun 12 16:44:28 2015] [info] Initial (No.1) HTTPS request received for child 195 (server demoflower.eatthismuch.com:80)
[Fri Jun 12 16:44:28 2015] [debug] mod_proxy_wstunnel.c(377): [client 71.189.228.118] canonicalising URL //127.0.0.1:5555/update-dashboard
[Fri Jun 12 16:44:28 2015] [debug] proxy_util.c(1509): [client 71.189.228.118] proxy: wss: found worker wss://127.0.0.1:5555/update-dashboard for wss://127.0.0.1:5555/update-dashboard
[Fri Jun 12 16:44:28 2015] [debug] mod_proxy.c(1020): Running scheme wss handler (attempt 0)
[Fri Jun 12 16:44:28 2015] [debug] mod_proxy_http.c(1978): proxy: HTTP: declining URL wss://127.0.0.1:5555/update-dashboard
[Fri Jun 12 16:44:28 2015] [debug] mod_proxy_wstunnel.c(659): [client 71.189.228.118] AH02451: serving URL wss://127.0.0.1:5555/update-dashboard
[Fri Jun 12 16:44:28 2015] [debug] proxy_util.c(2014): proxy: WSS: has acquired connection for (127.0.0.1)
[Fri Jun 12 16:44:28 2015] [debug] proxy_util.c(2070): proxy: connecting wss://127.0.0.1:5555/update-dashboard to 127.0.0.1:5555
[Fri Jun 12 16:44:28 2015] [debug] proxy_util.c(2196): proxy: connected /update-dashboard to 127.0.0.1:5555
[Fri Jun 12 16:44:28 2015] [debug] proxy_util.c(2447): proxy: WSS: fam 2 socket created to connect to 127.0.0.1
[Fri Jun 12 16:44:28 2015] [debug] proxy_util.c(2579): proxy: WSS: connection complete to 127.0.0.1:5555 (127.0.0.1)
[Fri Jun 12 16:44:28 2015] [info] [client 127.0.0.1] Connection to child 0 established (server demoflower.eatthismuch.com:80)
[Fri Jun 12 16:44:28 2015] [info] Seeding PRNG with 656 bytes of entropy
[Fri Jun 12 16:44:28 2015] [debug] mod_proxy_wstunnel.c(518): [client 71.189.228.118] sending request
[Fri Jun 12 16:44:28 2015] [debug] ssl_engine_kernel.c(1819): OpenSSL: Handshake: start
[Fri Jun 12 16:44:28 2015] [debug] ssl_engine_kernel.c(1827): OpenSSL: Loop: before/connect initialization
[Fri Jun 12 16:44:28 2015] [debug] ssl_engine_kernel.c(1827): OpenSSL: Loop: unknown state
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f0b6c001650 [mem: 7f0b68006f83]
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_kernel.c(1837): OpenSSL: Write: SSL negotiation finished successfully
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f0b60001650 [mem: 7f0b60006f83]
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_kernel.c(1837): OpenSSL: Write: SSL negotiation finished successfully
[Fri Jun 12 13:44:32 2015] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f0b70001650 [mem: 7f0b70006f83]
[Fri Jun 12 13:44:32 2015] [debug] ssl_engine_kernel.c(1837): OpenSSL: Write: SSL negotiation finished successfully
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f0b6c001650 [mem: 7f0b70006f83]
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_kernel.c(1837): OpenSSL: Write: SSL negotiation finished successfully
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f0b60001650 [mem: 7f0b70006f83]
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_kernel.c(1837): OpenSSL: Write: SSL negotiation finished successfully
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_io.c(1908): OpenSSL: I/O error, 5 bytes expected to read on BIO#7f0b64001650 [mem: 7f0b70006f83]
[Fri Jun 12 16:44:32 2015] [debug] ssl_engine_kernel.c(1837): OpenSSL: Write: SSL negotiation finished successfully

Any ideas what might be going wrong? I've tried a lot of different apache2 settings to no avail. Thanks for reading!

Answer 1

I have a similar apache config file and it was not working correctly and I saw the OpenSSL error 5 bytes expected message.

I added the following:

ProxyPreserveHost On

and now the application works correctly. I still see the OpenSSL error, but it does not appear to be a problem.

Note the declining http URL is just the normal operation of mod_proxy trying all lower level proxy modules in a specified order until it finds one that matches.

Also I am using ws: protocol not wss: to talk to Tomcat 8 backend.