Git Keeps Building on my 'known_hosts' File For Every Push

Question

I created a local git repository that I want to push on to both Github and also Amazon's CodeCommit.

I set up two remotes and named them accordingly:

git remote add github [email protected]:mygit/myrepo.git

and

git remote add codecommit ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/myrepo

The Github repo works fine. The CodeCommit repo works, but every time I run a push with:

git push -u codecommit master

it adds on to my '~/.ssh/known_hosts' file for every single push. If I continue I will have a million entries in my 'known_hosts' file.

The only difference I can see is the formatting on CodeCommit remote link, which is slightly different than the Github format, as shown above. When you create a repo on CodeCommit, it provides a remote link:

git clone ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/myrepo

Since I am pushing a local git repo on to CodeCommit, I removed the 'git clone' portion and replaced it with:

git remote add codecommit ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/myrepo

CodeCommit also requires that you create a '~/.ssh/config' file with:

Host git-codecommit.*.amazonaws.com
  User Your-IAM-SSH-Key-ID-Here
  IdentityFile ~/.ssh/Your-Private-Key-File-Name-Here

I created this file as instructed.

Anyone run into this issue while using two remotes: Github and CodeCommit?

Answer 1

It is probably that CodeCommit is behind a load balancer, and each node has its own SSH host key. As you hit different nodes, they are presenting you with their own host key, which your ssh client is caching in known_hosts.

You can just ignore this, but if you prefer not to have the keys accumulating, you can opt to ignore caching for this remote.

In ~/.ssh/config, just add a couple of parameters to the entry you have already added.

Host git-codecommit.*.amazonaws.com
  User Your-IAM-SSH-Key-ID-Here
  IdentityFile ~/.ssh/Your-Private-Key-File-Name-Here
  UserKnownHostsFile /dev/null
  CheckHostIP no

Answer 2

I don't recommend adding UserKnownHostsFile /dev/null to your SSH config. It does stop the caching of hosts but still requires you to accept the unknown host everytime. And if you use some git client's like SourceTree, this will auto-accept. At the end of the day you no longer actually checking who you are connecting to which is a security flaw! Since you are still using Public key authentication then it is difficult to falsify the end host but not impossible.

@Dan Lowe is correct that it is behind a load balancer and you will be redirected to different IP addresses each time you connect. I had the same discovery as you did with hundreds of known_hosts which all were to code repositories, github.com, bitbucket.org, gitlab.com, etc. The difference is with a commercial load balancer the host keys are all the same. Check your known_hosts file for that to be the case. github.com has this configuration.

Which means you can set a host key to match to the domain while ignoring the changing IP address. HostKeyChecking will still take place and you will have pre-approved the valid key so you won't be accepting a unknown host every time.

Recommendation (what I did)

1. Follow these instructions on this serverfault answer. It will help you to properly ssh-keyscan and add the host keys to known_hosts file. I recommend wiping out your old known_hosts file ($> mv known_hosts known_hosts.bkup && touch known_hosts) and only use the backup as reference for what you have connected with in the past.

   Note: there are different types of host_key_types but the most common/compatible and recommended is ssh-rsa at this time

   # known_hosts
   # each entry should have a domain host_key_type host_key
   github.com ssh-rsa AAAAB3NzaC1yc2E...A8VJiS5ap43JXiUFFAaQ==
   git-codecommit.*.amazonaws.com <host_key_type> <host_key>
   # If there is a specific port
   [domain]:2222 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbm...FEQVnnyjKaQ=
   

2. For future maintenance of known_hosts, I recommend placing the domain host keys from manual installation in step 1 into /etc/ssh/ssh_known_hosts for all users. This way these host keys are not hidden within your accepting of "The authenticity of host can't be established. Are you sure you want to continue to connect?" list when you audit yourself next time. Protect these manual verifications with:

   # This prevents tampering unless someone has root access to your system
   $> chmod 444 /etc/ssh/ssh_known_hosts
   $> sudo chown root:wheel /etc/ssh/ssh_known_hosts
   
   # leave your ~/.ssh/known_hosts as writable by you so that you may accept other SSH host keys in the future, unless you want to prevent this and only allow manual host verification
   $> ls -l ~/.ssh/known_hosts
   # -rw------- 1 username  staff  0  date  /home/username/.ssh/known_hosts
   

3. Add CheckHostIP no to your ~/.ssh/config for only those domain level host keys with known load balancers that you have added. I do NOT recommend doing setting this for Host * (all).

   Host git-codecommit.*.amazonaws.com
     User Your-IAM-SSH-Key-ID-Here
     IdentityFile ~/.ssh/Your-Private-Key-File-Name-Here
     CheckHostIP no
   

4. If you haven't, I would also make your SSH config read-only with chmod 440 ~/.ssh/config

FYI

If you have additional hosts (usually local) on your network that you regularly ssh to, use this opportunity to set their host keys as well. In /etc/ssh/ssh_known_hosts you can set the domain and the specific ip address (& port) for a host. The specific ip only works if you have a DHCP address reservation set up in your LAN routing table.

# LAN domain = <computer_name>.local or <computer_name>.<lan_domain>
computer_name.local,10.0.0.3 ssh-rsa AAAAB3NzaC1yc2EAAA...FVBoGqzHM9yXw==
[computer_name.local]:2222,[10.0.0.3]:2222 ssh-rsa AAAAB3NzaC1yc2EAAA...FVBoGqzHM9yXw==