I have a Docker Swarm cluster on AWS which I am trying to load balance using HAProxy. My setup which is behind a VPC looks similar to this:
haproxy_server 10.10.0.10
docker_swarm_master1 10.10.0.12
docker_swarm_master2 10.10.0.13
docker_swarm_worker3 10.10.0.14
My only Tomcat container is currently on master_1 and below is my current HAProxy config file:
global
log 127.0.0.1 local0
log 127.0.0.1 local0 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
maxconn 2000
frontend servers
bind *:80
bind *:8443 ssl crt /etc/haproxy/certs/ssl.pem
default_backend hosts
backend hosts
mode http
balance roundrobin
option httpchk OPTIONS /
option forwardfor
option http-server-close
server swarm 10.10.0.12:8443 check inter 5000
I am able able to see the index.html page in the webapps directory when I do the following from the HAProxy server:
curl -k https://10.10.0.12:8443/docs/index.html
However when I try the following curl command below, I get a 503 server not available error
curl -k https://10.10.0.10:8443/docs/index.html
Anyone know what I am doing wrong? I have spent half the day on this to no avail.
EDIT
curl -XOPTIONS -vk https://10.10.0.10:8443/docs/index.html
* Trying 10.10.0.10...
* Connected to 10.10.0.10 (10.10.0.10) port 8443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: *.secreturl.com (does not match '10.10.0.10')
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: OU=Domain Control Validated,CN=*.secreturl.com
* start date: Sat, 27 Jun 2016 16:39:39 GMT
* expire date: Tue, 11 Jun 2020 18:09:38 GMT
* issuer: C=US,ST=Arizona,L=Scottsdale,O=GoDaddy.com\, Inc.,OU=http://certs.godaddy.com/repository/,CN=Go Daddy Secure Certificate Authority - G2
* compression: NULL
* ALPN, server did not agree to a protocol
> OPTIONS / HTTP/1.1
> Host: 10.10.0.10:8443
> User-Agent: curl/7.47.0
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 503 Service Unavailable
< Cache-Control: no-cache
< Connection: close
< Content-Type: text/html
<
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Closing connection 0
curl -XOPTIONS -vk https://10.10.0.12:8443/docs/index.html
* Trying 10.10.0.12...
* Connected to 10.10.0.12 (10.10.0.12) port 8443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: *.secreturl.com (does not match '10.10.0.10')
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: OU=Domain Control Validated,CN=*.secreturl.com
* start date: Sat, 27 Jun 2016 16:39:39 GMT
* expire date: Tue, 11 Jun 2020 18:09:38 GMT
* issuer: C=US,ST=Arizona,L=Scottsdale,O=GoDaddy.com\, Inc.,OU=http://certs.godaddy.com/repository/,CN=Go Daddy Secure Certificate Authority - G2
* compression: NULL
* ALPN, server did not agree to a protocol
> OPTIONS / HTTP/1.1
> Host: 10.10.0.12:8443
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Allow: GET, HEAD, POST, PUT, DELETE, OPTIONS
< Content-Length: 0
< Date: Sat, 24 Dec 2016 18:39:27 GMT
<
* Connection #0 to host 10.10.0.12 left intact
If you get a
503 Service Not Available, then your health check fails.From your configuration, HAProxy will use
OPTIONS http://10.10.0.12:8443/which will fail: your backend accept HTTPS connections. To fix that, tell HAProxy to use HTTPS:Note: you can enable the stat page with
That should help you debug further issues.
Edit:
The stat page shows
L7STS/404, that's the http code HAProxy gets. HAProxy currently checkshttps://10.10.0.12:8443/while you testhttps://10.10.0.12:8443/docs/index.html. Perhaps you should use this url in your check: