Client 443/HTTPS -> Virtual IP -> HA Proxy 80/HTTP and 443/HTTPS SSL terminaison -> Varnish 81/HTTP -> 82/HTTPS and 442/HTTPS Apache

Question

At home, I have 1 hosted server with some websites, using Virtualmin (great tool!) and everything is working fine with HTTPS, HTTP...even behind my internet router, using NAT by forwarding ports 80 and 443 to my web server.

Client 443/HTTPS -> my Internet Router NAT -> 80/HTTP or 443/HTTPS Apache

I am doing some improvments by adding a load balancer HA Proxy in front of Apache : working fine.

Client 443/HTTPS -> my Internet Router NAT -> HA Proxy 443/HTTPS SSL terminaison -> 82/HTTP or 442/HTTPS Apache (192.168.0.40)

That means my websites are still available on HTTPS protocol but on port 442

And I even added keepalived and it's working fine :

Client 443/HTTPS -> my Internet Router NAT -> KEEPALIVED IP ADDRESS -> HA Proxy 443/HTTPS SSL terminaison -> 82/HTTP or 442/HTTPS Apache

Now, I would like to add the Web Caching Varnish. As you may know, the free version of Varnish does not support HTTPS. It's not a problem for me because I have HA Proxy that handles SSL terminaison. What I want to do is to set Varnish behind HA Proxy and in front of Apache.

Client 443/HTTPS -> my Internet Router NAT -> KEEPALIVED IP ADDRESS -> HA Proxy 443/HTTPS SSL terminaison -> Varnish 81/HTTP -> 82/HTTP or 442/HTTPS Apache

But it does not work... When I try to acccess to my website using HTTPS, for instance https://myexemple.com, the homepage is loaded but all other static files not... Looking into the page network inspector (F12 from the browser), I can see that these static files are served using HTTP (see attached screenshot).

enter image description here

Just installed wordpress for this website, nothing else.

Did somebody have a similar issue or maybe succceed in implementing an architecture schema like mine:

Client 443/HTTPS -> KEEPALIVED IP ADDRESS -> HA Proxy 443/HTTPS SSL terminaison -> Varnish 81/HTTP -> 82/HTTP or 442/HTTPS Apache

In general, how do you handle HTTPS when using Varnish? Especially with Wordpress: in the backend, we have to specify the URL of the website (i.e. https://kmx.ovh in my case)? Thank you

Well, I tried to setup Varnish behind HA Proxy and in front of my Apache websites, see configuration below :

Here is the configuration of my HA Proxy (quite simple):

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
        tune.ssl.default-dh-param 4096

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        retries 3
        option redispatch
        option http-server-close
        timeout connect 5s
        timeout client  50s
        timeout server  50s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


frontend https
        description "HA Proxy"
        bind *:80
        bind *:443 ssl crt /etc/haproxy/cert/
        mode http
        option httplog
        option forwardfor except 127.0.0.1
        http-request set-header X-Forwarded-Proto https
        http-response set-header MON-LOADBALANCER "HTTPS HA Proxy 40"
        default_backend varnish_pool

backend varnish_pool
        description 'Varnish backends pool'
        balance roundrobin
        mode http
        http-response set-header X-Frame-Options DENY
        http-response set-header X-XSS-Protection 1;mode=block
        http-response set-header X-Content-Type-Options nosniff
        http-response set-header Referrer-Policy no-referrer-when-downgrade
        default-server inter 15s fall 3 rise 2
        option httpchk GET / HTTP/1.1
        http-check expect status 200
        http-request set-header X-Forwarded-Port %[dst_port]
        server 40-80 192.168.0.40:81

And here is my Varnish configuration file (really basic...):

vcl 4.1;

backend default {
    .host = "192.168.0.40";
    .port = "82";
}

sub vcl_recv {
    set req.backend_hint = default;
    set req.http.Host = req.http.host;
}

sub vcl_backend_response {
    set beresp.http.MonX-Cache = "via Varnish cache 40";
}

sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    } else {
        set resp.http.X-Cache = "MISS";
    }
}

And the Varnish service:

[Unit]
Description=Varnish Cache, a high-performance HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/ man:varnishd

[Service]
Type=simple

# Maximum number of open files (for ulimit -n)
LimitNOFILE=131072

# Locked shared memory - should suffice to lock the shared memory log
# (varnishd -l argument)
# Default log size is 80MB vsl + 1M vsm + header -> 82MB
# unit is bytes
LimitMEMLOCK=85983232
ExecStart=/usr/sbin/varnishd \
          -j unix,user=vcache \
          -F \
          -a :81 \
          -T localhost:6082 \
          -f /etc/varnish/default.vcl \
          -S /etc/varnish/secret \
          -s malloc,256m
ExecReload=/usr/share/varnish/varnishreload
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

[Install]
WantedBy=multi-user.target

Answer 1

I advise you to add the following line to your HAProxy backend config:

http-request add-header X-Forwarded-Proto https if { ssl_fc }

While you already added an X-Forwarded-Port header, the X-Forwarded-Proto header is a more supported header for URL building.

A web application that is directly exposed to the end-users knows what the client URL scheme (HTTP or HTTPS) was and builds the URLS accordingly.

When sitting behind a caching proxy, modern web applications also support X-Forwarded-Proto to know which URL scheme they need to use. Despite Varnish only sending plain HTTP requests to your application, the initial connection could be over HTTPS.

That's why you need X-Forwarded-Proto. And to ensure that Varnish caches HTTP and HTTPS versions of the page separately, you have to add the Vary: X-Forwarded-Proto response header to the HTTP response that your application returns.

Debugging

An easy way to debug whether the X-Forwarded-Proto header is properly attached is by running varnishlog.

I suggest you run the following command:

sudo varnishlog -g request -q "reqHeader:X-Forwarded-Proto eq 'https'"

This command will show all incoming requests that contain an X-Forwarded-Proto with https as a value, referring to the HTTPS requests that HAProxy received and converted into plain HTTP requests to Varnish.

If it turns out your HTTPS requests don't contain that header, you still need to properly configure HAProxy.

If it turns out the X-Forwarded-Proto:https is set, but your application still returns plain HTTP responses, we need to look at your application.

Please confirm what your seeing so I can advise you on next steps.