Can't PURGE entire domain in Varnish, but can PURGE individual pages, do I have a misconfiguration?

1.8k views Asked by At

I'm assuming that I must have made a mistake in my Varnish configuration. (I'm running version 6.0.7.) Here is the relevant section:

####----SECTION THREE: PURGE RULES----####
# Access Control List to define which IPs can purge content
acl purge {
        "localhost";
        "127.0.0.1";
        "<my home IP address>";
        "::1";
}

####----SECTION FOUR: PROCESSING RULES----####
sub vcl_recv {

# Do not allow purging from non-approved IPs
    if (req.method == "PURGE") {
        if (!client.ip ~ purge) {
            return (synth(405, "This IP is not allowed to send PURGE requests."));
}
    return (purge);
}

# Allow banning regexes
    if (req.method == "BAN") {
        if (!client.ip ~ purge) {
            return (synth(405, "This IP is not allowed to send BAN requests."));
    }
        ban("req.http.host == " + req.http.host + " && req.url ~ ^" + req.url);
        return (synth(200, "Ban added"));
    }
# Allow purging regexes
        if (req.method == "PURGE") {
        if (req.http.X-Purge-Method == "regex") {
        ban("req.url ~ " + req.url + " && req.http.host ~ " + req.http.host);
        return (synth(200, "Banned"));
    } else {
    return (purge);
    }     

(N.B.: In my setup, Varnish is running on port 6081, with HAProxy in front of it.)

Using this setup, from either my home IP address or the server Varnish is running on, I am able to clear individual pages from the Varnish cache by running, e.g.:

curl -i -X PURGE https://example.com/page/

I can also clear the entire domain from the Varnish cache by running, e.g.:

curl -i -X BAN https://example.com

Likewise, I can clear individual pages from the Varnish cache by running, e.g.:

curl -i -X PURGE http://<IP of Varnish server>:6081/page/ -H "Host: example.com”

I can also clear the entire domain from the Varnish cache by running, e.g.:

curl -i -X BAN http://<IP of Varnish server>:6081 -H "Host: example.com”

But I have noticed that I cannot clear the entire domain by running either:

curl -i -X PURGE https://example.com

or:

curl -i -X PURGE http://<IP of Varnish server>:6081 -H "Host: example.com”

To clear the entire domain, I have to use BAN instead. I do not know if this is a problem or a misconfiguration, but it seems to be, because although I have read that PURGE is not used to clear everything, the Wordpress plugin I am using seems to be trying to, by sending a PURGE request to domain.com/.* when I hit the “Clear ALL Varnish Cache” button.

This, frankly, confuses me. Because it doesn’t work either in the plug-in itself or if I try it from the command line. What am I missing? Is the plugin just out of date? Did Varnish used to be able to clear the entire cache this way? I'd appreciate any pointers. Thanks!

1

There are 1 answers

1
Thijs Feryn On

Here's a cleaned up version of the purging logic in your VCL:

acl purge {
        "localhost";
        "127.0.0.1";
        "::1";
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (!client.ip ~ purge) {
            return (synth(405, "This IP is not allowed to send PURGE requests."));
        }
        if (req.http.X-Purge-Method == "regex") {
            ban("obj.http.x-url ~ " + req.url + " && obj.http.x-host ~ " + req.http.host);
            return (synth(200, "Banned"));
        }
        return (purge);
    }
}

sub vcl_backend_response {
    set beresp.http.x-url = bereq.url;
    set beresp.http.x-host = bereq.http.host;
}

sub vcl_deliver {
    unset resp.http.x-url;
    unset resp.http.x-host;
}

Purging

The PURGE HTTP method is what triggers the cache invalidation. Without any headers return(purge); will be called, which will invalidate the exact URL that was used.

If the X-Purge-method request header is added, and is set to regex, a regular expression match will occur. Invalidating multiple objects using a regular expression cannot be done with return(purge);, and requires banning.

Bans

The ban() function adds a ban expression to the ban list. Expressions on this list are matched to all the objects that are stored in cache. The matching ones are removed.

You can consult the ban list using the following command on your Varnish server:

varnishadm ban.list

Lurker-friendly bans

The ban lurker, a special thread that processes the ban list, doesn't have access to the request context. If you want this thread to remove the objects from cache, based on request parameters like the URL or the hostname, you need to apply som trickery.

As you can see, the pattern I suggest, matches on obj.http.x-url & obj.http.x-host. These are set in vcl_backend_response via 2 custom response headers.

If you don't do this, the ban lurker cannot match the objects, and the responsibility will shift to the next request. Even without these so-called lurker-friendly bans, banning will still work, but it is not as efficient.

Testing it

The first example will invalidate http://example.com/page from the cache

curl -i -XPURGE -H "Host: example.com" "http://localhost:6081/page/"

The next example will invalidate all objects whose URL starts with http://example.com/page:

curl -i -XPURGE -H "Host: example.com" -H "X-Purge-Method: regex" "http://localhost:6081/page/"

The last example will invalidate all objects from cache for the example.com domain:

curl -i -XPURGE -H "Host: example.com" -H "X-Purge-Method: regex" "http://localhost:6081/.*"

Here's the ban list item for that last example:

root@varnish:/etc/varnish# varnishadm ban.list
Present bans:
1606393598.913178     0 -  obj.http.x-url ~ /.* && obj.http.x-host ~ example.com
1606393484.785268     3 C

As you can see, it matches all URLs for the example.com host. When you perform an HTTP request after that, you'll see the Age: 0 response header. This indicates that this response has been stored in cache for zero seconds. This means the ban was successful.