AWS EC2 `ssh: connect to host ec2-{ip}.compute-1.amazonaws.com port 22: Connection refused` only if `user-data` uses `systemctl` commands

Question

AWS Setup

I have a AWS EC2 instance running with:

• t4g.nano
• al2023-ami-2023.3.20240122.0-kernel-6.1-arm64 (AL2023)
• a Security Group setup to allow SSH from my personal IP address
• sshd is installed & running
• a valid ssh key/value pair assigned
• a Public IP address assigned (so I can connect to it)

Local Setup

I am running on MacOS v13.6.2 with ssh:

$ ssh -V 
OpenSSH_9.0p1, LibreSSL 3.3.6

Problem

With No User Data

If my EC2 instance does not have any user-data, I am able to connect to the instance using ssh. Example ssh:

$ ssh -i "my-key-pair.pem" ec2-user@ec2-{ip}.compute-1.amazonaws.com

This connects just fine.

With User Data

If my EC2 instance has a user-data script with certain commands, then I get the following error:

$ ssh -vvv -i "my-key-pair.pem" ec2-user@ec2-{ip}.compute-1.amazonaws.com   

OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/dj/.ssh/config
debug1: /Users/dj/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/dj/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/dj/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to ec2-{ip}.compute-1.amazonaws.com port 22.
ssh: connect to host ec2-{ip}.compute-1.amazonaws.com port 22: Connection refused

It is difficult to track down what commands causes this issue, but here is what I know so far:

• If my script contains anything using systemctl, then it causes the ssh connection to be refused:

  # example, I install docker in `user-data`
  yum install -y docker
  
  service docker start
  
  systemctl enable docker.service
  systemctl enable containerd.service
  

  • This will cause the connection refused.

Question

I am not an ssh or scripting expert. Why does my user-data script's call to systemctl suddenly cause my ssh to break?