AWS Spot Instance bootstrap failed : package amazon-ssm-agent is already installed

Question

I have a setup spot instance that must install docker on bootstrap. Lately, I am facing a new issue during the dependencies download phase.

The download transaction is failing as follows :

Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
--------------------------------------------------------------------------------
Total                                               78 MB/s | 131 MB  00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
Transaction couldn't start:
package amazon-ssm-agent-3.1.1080.0-1.x86_64 (which is newer than amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64) is already installed
file /etc/amazon/ssm/README.md from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /etc/amazon/ssm/NOTICE.md from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /etc/amazon/ssm/RELEASENOTES.md from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /etc/amazon/ssm/amazon-ssm-agent.json.template from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /usr/bin/amazon-ssm-agent from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /usr/bin/ssm-agent-worker from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /usr/bin/ssm-cli from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /usr/bin/ssm-document-worker from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /usr/bin/ssm-session-logger from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64
file /usr/bin/ssm-session-worker from install of amazon-ssm-agent-3.0.1124.0-1.amzn2.x86_64 conflicts with file from package amazon-ssm-agent-3.1.1080.0-1.x86_64

...

Redirecting to /bin/systemctl start docker.service
Failed to start docker.service: Unit not found.

Apparently the Transaction is failing due to a conflict between two versions of amazon-ssm-agent !! Due to this error the install for the other dependencies is aborted.

Is it possible to fix one version or avoid the install of a new version of SSM ?

Answer 1

Instead of uninstalling amazon-ssm-agent, you can skip it and it will should auto-update.

sudo yum update -y --exclude=amazon-ssm-agent

I had a similar problem and it looks like amazon-ssm-agent is automatically updated on startup which can conflict with the yum update. On my "good" servers, yum update completes before the ssm agent auto-updater starts.

We do a ps -aux at the end of UserData to show running processes, and on a server that successfully starts it shows that /var/lib/amazon/ssm/update/amazon-ssm-agent-updater/3.1.1188.0/updater is running. By this point, yum has finished.

I updated my UserData to exclude amazon-ssm-agent from the update and the server had no problem after that. After I let the server "stabilize" for a few minutes, I checked the version (yum info amazon-ssm-agent) and it said v3.1.1260.0 which is the current as of this post. So, it launches with a non-current version then auto-updates to the current version, without the help of yum.

Answer 2

The solution provided by @Riz in the comments solved the issue.

I have to erase ssm on bootstrap :

sudo yum erase amazon-ssm-agent --assumeye && sudo yum update -y
sudo yum install -y docker && sudo service docker start

Answer 3

I was facing similar errors - installing an unrelated yum package on AL2 ECS optimized AMI (comes with SSM agent already installed) would ~10% of the time fail with error:

Error: Rpmdb checksum is invalid: pkg checksums: amazon-ssm-agent-0:3.1.1732.0-1.amzn2.x86_64

AWS Support said to see https://access.redhat.com/solutions/3573491 (account required)

The checksum in yumdb was getting corrupted. Since multiple packages were installed/updated during initialization script, the yum database was getting updated a lot. The fix was to ensure you do yum install -y yum-utils at the beginning, where you are likely doing yum update and then, right before you were facing the problem, run yumdb sync-force. That fixed the problem entirely, and in the output you can see some information about ssm versions.

This method avoids uninstalling SSM and then re-installing it later.

The solution offering sudo yum update -y --exclude=amazon-ssm-agent did not work for me and still exhibited about a 10% error rate.