How to get AWS command line interface to work in cygwin

Question

I installed the AWS command line interface on my Windows 7 box, and it worked immediately when I called commands from a DOS shell.

But DOS, the worst language ever invented, is hideous for any serious scripting. So, I would like to to use the AWS CLI from bash via cygwin.

In my case, the installed AWS CLI is the Windows version. In principle, that should not be a problem because Windows commands are executable from cygwin. (cygwin includes your Windows environmental variables, such as PATH, in its own environment.)

Unfortunately, when I first tried to execute an AWS CLI command from cygwin/bash, I got an error:

$ aws s3 cp code.tgz s3://xyz/
upload failed: .\code.tgz to s3://xyz/code.tgz
Unable to locate credentials

This error is likely because the AWS CLI is looking in the wrong directory for the credentials file. On Windows, it expects that file to be in %UserProfile%.aws and in unix in ~/.aws.

One hack work around is that in my home directory I created a new file named config_credentials which contains a union of the contents of that directory's files config and credentials. I then made a new Windows System env var named AWS_CONFIG_FILE whose value is the path to config_credentials. Success: AWS CLI commands issued from cygwin now work.

I am wondering if there is a better solution?

I am curious why AWS CLI initially failed to search in the correct home directory for the config and credentials files. I also wonder if there is a way to correct that (this would eliminate the need for the AWS_CONFIG_FILE env var).

Answer 1

Here's a way that jumps through all the Microsoft hoops; it's tedious to do for setup, but works very cleanly once you're actually using the AWS CLI. You don't need to do anything weird in Cygwin until the very end, but you may need Windows admin privs depending on your computer policies.

1. Install the Windows version of the AWS CLI. (It's currently distributed as an .msi installer, so likely needs admin privs to install to %PROGRAMFILES%.)

2. Launch a good ol' CMD.EXE window. Do the aws configure step to make sure the Windows AWS CLI is installed and working, and that your Windows .aws directory is created and populated with files.

3. cd around into your Cygwin home directory, but still in the CMD.EXE window.

4. If you have a .aws in your Cygwin $HOME, delete it or move it out of the way.

5. mklink /j .aws %USERPROFILE%\.aws This creates an NTFS directory junction, which is halfway between a Unix hard link and a Unix symbolic link, works only for directories, and -- unfortunately -- is a built-in command to CMD.EXE, not an external binary of its own. This makes a sort-of-directory named .aws in your Cygwin $HOME, linked to your Windows "home". (This command is why you had to run CMD.EXE. There's probably a way to do it inside of PowerShell, and I don't care.)

6. Type dir to see that the .aws junction exists and how CMD.EXE displays it.

7. Exit CMD.EXE and wash your hands thoroughly with soap. You won't need to touch that again.

8. Start a Cygwin shell, of whatever combination you prefer. (mintty+bash, or whatever) Run ls -lFd .aws to see the newly-created junction -- note how the Cygwin DLL presents directory junctions as symlinks.

9. You may need to adjust your PATH inside Cygwin to pick up the AWS CLI installation.

With the junction in place, using aws inside my Cygwin shell performs normally, and all tools are using the same config/credentials files, right where the (Windows) AWS CLI v2 installation expects to find them.

The one minor hitch is that AWS CLI under Windows uses CRLF newlines in all its output, so if doing any scripting with aws [...] --output text, you need to pipe that through tr -d '\r' before doing anything with the results. Else you get the hidden carriage return stuck on the end of your lines, fouling things up at random times. (You can pipe AWS output through cat -A to make them visible for debugging.)

Answer 2

After a LOT of time spent on this, I found a solution that works.

The primary issue is that the cygwin didn't come with python installed, and doesn't know where to find the existing Windows Anaconda version on your machine. This can be verified by running which python from within cygwin - it couldn't find where python is saved. Note that this can be confusing because running pip install awscli likely doesn't throw an error message. Cygwin actually installs awscli in the Window's Anaconda installation of Python (I find this odd since we didn't run conda install awscli).

HOWEVER, rather than try to point cygwin to the already installed version of Anaconda python on your machine it will save you a ton of headache to just install a cygwin-specific instance of python. The steps to do so are documented here: http://wiki.fast.ai/index.php/Awscli_in_cygwin)

1. pip uninstall awscli
2. wget rawgit.com/transcode-open/apt-cyg/master/apt-cyg
3. install apt-cyg /bin
4. apt-cyg install python
5. wget https://bootstrap.pypa.io/get-pip.py
6. python get-pip.py
7. pip install awscli

...Note, however, that the first command pip uninstall awscli "hung up" for me. So just escape out of it using quit() and continue with the others in order.

You can check that everything worked if you run which python in cygwin and it points to the cygin version (i.e. /usr/bin/python , as opposed to: /users/.../Anaconda2/).

Additionally, if you happen to be asking this in conjunction with watching the setup video for the fast.ai course (http://course.fast.ai/lessons/aws.html), then the next step is CRITICAL for Windows users: when you download all the shell scripts from Github setup folder (https://github.com/fastai/courses/tree/master/setup), Windows automatically adds CRLF line terminators! Therefore, in cygwin, run the following commands to remove these line endings:

1. apt-cyg install dos2unix
2. dos2unix setup_p2.sh
3. dos2unix setup_instance.sh
4. then finally, bash setup_p2.sh

This should do the trick.

Answer 3

I had the same problem. I got around it by installing a new copy of AWSCLI within Cygwin. You first need to install the "curl" and "python" Cygwin packages, then you can install AWSCLI as follows:

$ curl -O https://bootstrap.pypa.io/get-pip.py
$ python get-pip.py
$ pip install awscli

If you're running bash, and you've previously executed the Windows AWS command line, you need to clear the cached path as follows:

$ hash -d aws

"aws --version" will then look similar to this:

aws-cli/1.8.1 Python/2.7.10 CYGWIN_NT-10.0/2.2.1(0.289/5/3)

as opposed to the Windows command line output, which looks similar to this:

aws-cli/1.8.1 Python/2.7.9 Windows/8

I'm now able to do "aws configure" under Cygwin, and everything works as it should.

Answer 4

In my case, I had to:

# Run Cygwin64 Terminal by right-clicking on the icon and selecting "Run as Administrator"
pip3.7 install awscli
aws configure

Once my AWS credentials had been configured I could run awscli commands on Cygwin where the latter had been launched normally, i.e. as a non-Administrator user.

Earlier I attempted the same steps but I launched Cygwin as non-Adminstrator and the awscli installation through pip didn't take. I didn't save the error messages, unfortunately.

Answer 5

I used aws configure from DOS cmd window to create the cfg files (config and credentials) and tested them with a sample aws cmd (in DOS window). Then I copied Users\.aws folder to spot where cygwin thinks the user home folders are (in my case c:\cygwin64\home\). I then used TextPad to convert the line endings (use file>save-as; select unix line endings; make sure the files don't get renamed x.txt). Now it works.