AWS S3 Requester Pays Does not work with TransferManager

Question

I am trying to use TransferManager class in my java program to download a file from a Requester Pays Buckets. I am getting "Status Code: 403" exception from amazon AWS. I wrote a simple program to test this feature and compare it single connection way of downloading a file. Here is my code:

import java.io.*;
import com.amazonaws.auth.*;
import com.amazonaws.services.s3.*;
import com.amazonaws.services.s3.model.*;
import com.amazonaws.services.s3.transfer.*;
import com.amazonaws.util.IOUtils;
public class RequesterPaysTest {
    final static AWSCredentials awsCredentials = new BasicAWSCredentials(MY IAMAccessKey,MY IAMSecretKey);
    final static String bucketName = "7268982505fe.mixnode.com";
    final static String fileName = "5379-7268982505fe-0-1496081968663.warc.gz";
    final static AWSCredentialsProvider awsCredentialsProvider =  new AWSStaticCredentialsProvider(awsCredentials);
    final static AmazonS3 s3client = AmazonS3ClientBuilder.standard().withCredentials(awsCredentialsProvider).withRegion(MY bucketRegion).build();
    final static GetObjectRequest getRequest = new GetObjectRequest(bucketName, fileName, true);

    static void testSimpleRequesterPays() {
        try {
        S3Object object = s3client.getObject(getRequest);
        InputStream objectData = object.getObjectContent();
        FileOutputStream out = new FileOutputStream (new File(fileName));
        IOUtils.copy(objectData, out);
        out.close();
        System.out.println(" Simple RequesterPays successful");
        } catch (Exception e) {
        System.out.println(" Simple RequesterPays unsuccessful: " + e.getMessage());
        }
    }
    static void testTransferManagerRequesterPays() {
        try {
            TransferManager tx = TransferManagerBuilder.standard().withS3Client(s3client).build();
            Download download = tx.download(getRequest, new File(fileName));
            while (download.isDone() == false)
                Thread.sleep(10);
            System.out.println(" TransferManager RequesterPays successful");
        } catch (Exception e) {
        System.out.println(" TransferManager RequesterPays unsuccessful: " + e.getMessage());
        }
    }
public static void main(String[] args) throws IOException {
    testSimpleRequesterPays();
    testTransferManagerRequesterPays();
}
}

And here is the output:

Simple RequesterPays successful
 TransferManager RequesterPays unsuccessful: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 77D5EBF5EE195A7A)

As you can see, the same file can be downloaded using simple method but not with TransferManager. I tried to create my own bucket and played around with permissions but it did not work. I was wondering if I miss anything in my code? Or whether AWS s3 does not have support for using TransferManager on a requester pays bucket?

Answer 1

TransferManager internally using http HEAD method instead of GET for downloading objects(file(s)). You can find it by debugging TransferManager SDK code. If you don't have HEAD method configuration in your aws api gateway, you have to create HEAD method configuration to match with TransferManager SDK code. By the way if resource not found it should give 404 but you are getting 403. reason is here https://forums.aws.amazon.com/thread.jspa?threadID=216684