Problems with redirecting to signed cloud storage URL (cURL?)

Question

I am creating a Firebase HTTP function that uploads a file to Cloud Storage, creates a signed URL to the file, and then redirects the client to that URL. Using Postman with automatic redirect following turned on, the file is retrieved correctly. However, if I try to turn on redirects while using cURL (curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}'), the following error is returned by Cloud Storage:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
    <StringToSign>GET

application/json
1602245678
/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>

If I make the request with form encoded data instead, it works in cURL as well: curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"

If I try to manually make a GET request to the URL in Postman, I get an equivalent error:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
    <StringToSign>GET


1602246219
/www.google.com/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>

If I paste the URL into a browser or use cURL to download the signed URL, the file is also downloaded correctly.

I am using the following function to get the signed url:

async getSignedUrl(file: File, expireAt: number): Promise<string> {
    const [url] = await file
        .getSignedUrl({
            action: "read",
            expires: expireAt
        });

    return url
}

which returns a signed URL in the following format: https://storage.googleapis.com/example.appspot.com/exampleBucket/exampleFile.txt?GoogleAccessId=[Access ID]&Expires=1602246219&Signature=[Signature] (I've noted that the value of "Expires" is the same value returned in the tag).

My suspicion is that Postman and cURL adds something to the request which results in a different signature, but I am not sure exactly what is going on.

What is happening when letting cURL follow the redirect or when creating a GET request in Postman, that leads to this difference in signature?

Answer 1

Joss Barons answer helped me in the right direction, but it is not true that the Content-Type has to be application/octet-stream. That is only used for creating a signed url that can be used for uploading a file. In my case, when creating the signed url using the Cloud Storage SDK for node, I didn't specify a Content-Type, so when sending a GET request to the signed url, it must not contain a Content-Type header.

Answer 2

If I understood correctly, the issue arises in two scenarios

1. When hitting your CF through curl with

curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}')

According to the example in github in the docs Signed URL v4, 'Content-Type: application/octet-stream' should be used:

curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '${url}'

I tried with the following with successfully result:

curl -X PUT -H 'Content-Type: application/octet-stream' -d '{"example": true}' 'https://storage.googleapis.com/...'

If I try with the content-type you shared with failed results.

2.

If I try to manually make a GET request to the URL in Postman, I get an equivalent error: I tried a simple GET in postman using a Signed URL and it worked just fine

Command used in gsutil to get the signed URL:

 gsutil signurl -d 10m key.json gs://BUCKET/aa.png

Then I tried a GET on postman and worked just fine.

I also tried with a Signed URL to upload a File in Postman and worked just fine.

My thoughts are that, according to Common MIME types

application/octet-stream is the default value for all other cases (not textual files).

When you set the content type as application/json you specify a JSON format, but not an object or file. That's why it works with the following, since you are not specifying the header content-type, the default is taken application/octet-stream

curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"