I have a bucket with enabled encryption configured:
- encryption key type: AWS Key Management Service key (SSE-KMS)
- using AWS managed key (aws/s3). Bucket encryption config:
I have nodejs project using streams to upload/download files on S3 bucket. The problem is that I can make successful upload, but when I try to download/get the file using stream it only emits 'data' event once and after that emits 'end' event, but there is still data to be consumed as you can see from the logs below. Do you have any idea why this is happening?
Additional info:
- When I upload to bucket where I don't have encryption enabled it works fine.
- My policy has "kms:Decrypt"
- I've tried using aws cli to upload and download and it was successful.
- We create the file URL with s3.getSignedUrl('getObject', params)
- Using request.js
Download stream:
async downloadStream() {
logger.info('Getting download stream...');
const url = await this.getFileUrl(); // getSignedUrl
const headers = {};
this.downloadStream = request({ url, headers });
this.downloadStream.on('data', (chunk) => this.emit('data', chunk.length));
this.downloadStream.on('end', () => this.emit('end'));
this.downloadStream.on('error', (error = {}) => this.emit('failure', error));
logger.info('Acquired download stream successfully');
this.downloadStream.pipe(this.response);;
}
Logging info:
Full Transfer Statistics:
Bytes Transferred: 4491 of 13170 Bytes
Left: 8679
Average Chunk Size: 4491 bytes
Estimated Chunks Left: 2
Average Speed: 0.000260375 MB/s
Total Transfer Time: 13.127 seconds
Objects that are encrypted on server side with KMS requires to use SigV4 protocol when accessing them. source: aws forum
When creating AWS.S3 client add signatureVersion: 'v4' in the options:
This fixed my problem.