Varnish proxy for s3 bucket fails to sign REST calls with parameters in URL

Question

Im implementing Varnish HTTP proxy to serve content from S3 bucket. The content is very big JS client 2K+ files. I found out this guide https://info.varnish-software.com/blog/using-varnish-cache-secured-aws-s3-gateway

In general it signs http requests and redirects them to s3 endpoint

sub vcl_backend_fetch
{
  set bereq.http.Host = "bucket.s3.amazonaws.com";
  set bereq.http.Date = now;

  set bereq.http.NL = {"
"};

  set bereq.http.Authorization = "AWS YOURVERYLONGACCESSID:" +
    digest.base64_hex(digest.hmac_sha1("somelongvalueforsecret",
      bereq.method + bereq.http.NL + bereq.http.Content-MD5 + bereq.http.NL +
      bereq.http.Content-Type + bereq.http.NL + bereq.http.Date + bereq.http.NL + 
      "/" + "bucket" + bereq.url
    ));

  unset bereq.http.NL;
}

It work for retrieving files, however if i have http parameters in url i want to fetch it fails with error SignatureDoesNotMatch.

Answer 1

The code that i was using was outdated. I needed to sign my requests with sig4 version of AWS REST API. This repo was helpful:

https://github.com/xcir/libvmod-awsrest

IMPORTANT: be carefull if you url contain special symbols @ & [] etc. In my case signature was failing because i had [ and ] symbols in name of the files. Ive replaced them

set bereq.url = regsuball(bereq.url,"\[","%5B");
set bereq.url = regsuball(bereq.url,"\]","%5D");

and it worked.

Answer 2

The blog post from my colleague Reza dates back to 2016. Chances are that AWS has already changed the signing algorithm.

We have updated VCL code that works on AWS S3 right now, but it's part of our enterprise offering, and uses some of our enterprise VMODs.

If you're interested, you can spin up one of our official Varnish Enterprise images on AWS, which doesn't require you to purchase a license in advance. See https://docs.varnish-software.com/varnish-cache-plus/vmods/aws/ for more information.

But if you insist on doing this using Varnish Cache, I suggest you have a look at the S3 API spec, and adjust the signing yourself.