AWS Tools: Copy-S3Object script fails in 2.x with Error "Bucket does not exist"

4.3k views Asked by At

I am attempting to figure out why a script that works in AWS tools 1.x (I think 1.1.16?) Is now not working after upgrade to the latest AWS tools (2.0.3)

The Script

Import-Module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"

$creds = New-AWSCredentials -AccessKey [REDACTED] -SecretKey [REDACTED] 

Set-AWSCredentials -Credentials $creds

$a = Get-Content C:\users\killeens\desktop\temp\AmazonKeysToDownload.txt
$startingpath = "G:\TheFiles\"

$a | ForEach-Object {
    $keyname = $_

    $fullpath = $startingpath + $keyname
    write-host "fullpath: "$fullpath
    Get-S3Bucket -BucketName OURBUCKETNAME | Get-S3Object -Key $_ | Copy-S3Object -Key $keyname -LocalFile $fullpath

    }

The Problem

In 1.1.16, this works fine.

Now, under deadline in 2.0.3, I get the following error:

Copy-S3Object : The specified bucket does not exist

These details might be important

  • For what it's worth, our bucket name is all capital letters. ("COMPANYCLIENT")
  • This literally worked on my machine an hour or so ago. I then wanted to do something in parallel, so I downloaded powershell v4 and the latest AWS Tools. This problem kept happening. I have since reverted to powershell 3 but the issue remains.
  • I have not been able to find an old version of amazon 1.x tools to test

Troubleshooting so far

  • if I only execute Get-S3Bucket OURBUCKETNAME, it works
  • if I execute the script, leaving off the piped Copy-S3Object command, it works, outputting all of the objects that I imported in my file.
  • I checked and it doesn't appear that there is a BucketName parameter on the Copy-S3Object command according to the intellisense. If I try to specify one, I get an error.
1

There are 1 answers

0
SeanKilleen On BEST ANSWER

It appears there is also a cmdlet called Read-S3Object that ends up with the same result. Had to use that.

Didn't see anything about Copy-S3object being deprecated or having its functionality changed, so that's unfortunate.

Assuming you have:

  • Powershell V3
  • Amazon Tools for Powershell v2.x
  • Appropriate Amazon Credentials

Then the following script should work:

Import-Module "C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell\AWSPowerShell.psd1"

### SET ONLY THE VARIABLES BELOW ###

$accessKey = ""                                  # Amazon access key.  
$secretKey = ""                                  # Amazon secret key.
$fileContainingAmazonKeysSeparatedByNewLine = "" # Full path to a file, e.g. "C:\users\killeens\desktop\myfile.txt"
$existingFolderToPlaceDownloadedFilesIn = ""     # Path to a folder, including a trailing slash, such as "C:\MyDownloadedFiles\" NOTE: This folder must already exist.
$amazonBucketName = ""                           # the name of the Amazon bucket you'll be retrieving the keys for.

### SET ONLY THE VARIABLES ABOVE ###

$creds = New-AWSCredentials -AccessKey $accessKey -SecretKey $secretKey
Set-AWSCredentials -Credentials $creds

$amazonKeysToDownload = Get-Content $fileContainingAmazonKeysSeparatedByNewLine
$uniqueAmazonKeys = $amazonKeysToDownload | Sort-Object | Get-Unique
$startingpath = $existingFolderToPlaceDownloadedFilesIn

$uniqueAmazonKeys | ForEach-Object {
    $keyname = $_

    $fullpath = $startingpath + $keyname
    Read-S3Object -BucketName $amazonBucketName -Key $keyname -File $fullpath

    }

Obviously there would be better ways to produce this (as a function that accepts parameters, in a Powershell v4 workflow with parallel loops and a throttle count, better dealing with credentials, etc.) but this gets it done in its most basic form.