My terraform is spinning up two instances, and deploying the same two ansible playbooks on both:
resource "aws_instance" "X360Instance" {
count = var.MultiNode ? 2 : 1
ami = var.AMIID
instance_type = var.InstanceType
key_name = aws_key_pair.MasterKey.key_name
associate_public_ip_address = true
vpc_security_group_ids = [aws_security_group.X360SG.id]
subnet_id = aws_subnet.Public.id
root_block_device {
volume_type = "gp2"
volume_size = 100
}
###Assign a Name tag to the target machine instance. This tag is referenced by passed_in_hosts.
tags = {
Name = "X360tf"
App = "Node${count.index}"
}
###Wait for the route table to be available
depends_on = [aws_route_table_association.PublicRTA]
###Connect to the target machine using the keys in the local .ssh directory
connection {
type = "ssh"
user = var.Secrets ? "${local.secrets.TargetUser}" : var.TargetUser
private_key = file(pathexpand("~/.ssh/id_rsa"))
host = coalesce(self.public_ip, self.private_ip)
}
}
resource "null_resource" "ExecuteAnsible" {
count = var.MultiNode ? 2 : 1
provisioner "remote-exec" {
connection {
host = "${element(aws_instance.X360Instance.*.public_ip, count.index)}"
#host = coalesce(aws_instance.X360Instance[count.index].public_ip, aws_instance.X360Instance[count.index].private_ip)
user = var.Secrets ? "${local.secrets.TargetUser}" : var.TargetUser
private_key = file(pathexpand("~/.ssh/id_rsa"))
}
inline = ["echo 'connected!'"]
}
###Execute the PostgresInstaller ansible playbook on the target machine
provisioner "local-exec" {
command = <<EOF
ansible-playbook --extra-vars "@AnsiblePlaybooks/Variables.json" AnsiblePlaybooks/PostgresInstaller.yml
EOF
}
provisioner "local-exec" {
command = <<EOF
ansible-playbook --extra-vars "@AnsiblePlaybooks/Variables.json" AnsiblePlaybooks/X360Installer.yml
EOF
}
}
The two instances are deployed and both playbooks are executed on both. I can reach the application that was deployed by the ansible playbooks on both ec2 instances, so I know the playbooks executed correctly. However, at the end, I get the message saying 0 failures, but instead of showing the outputs, it fails to this error:
null_resource.ExecuteAnsible[1] (local-exec): PLAY RECAP *********************************************************************
null_resource.ExecuteAnsible[1] (local-exec): 3.83.34.169 : ok=51 changed=34 unreachable=0 failed=0 skipped=5 rescued=0 ignored=5
null_resource.ExecuteAnsible[1] (local-exec): 3.83.94.204 : ok=51 changed=34 unreachable=0 failed=0 skipped=5 rescued=0 ignored=5
null_resource.ExecuteAnsible[1]: Creation complete after 7m57s [id=7711336021038131401]
╷
│ Error: local-exec provisioner error
│
│ with null_resource.ExecuteAnsible[0],
│ on instances.tf line 226, in resource "null_resource" "ExecuteAnsible":
│ 226: provisioner "local-exec" {
│ Error running command ' ansible-playbook --extra-vars "@AnsiblePlaybooks/Variables.json" AnsiblePlaybooks/PostgresInstaller.yml
│ ': exit status 2. Output: ies.", "Unable to read consumer identity", "", "This system is not registered to Red Hat Subscription Management. You can use
│ subscription-manager to register.", "", "Waiting for process with pid 5991 to finish.", "Last metadata expiration check: 0:00:21 ago on Tue 31 May 2022 10:02:50 PM
│ UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version
│ Repository Size", "================================================================================", "Installing:", " createrepo_c x86_64
│ 0.17.7-3.el8 @commandline 89 k", "", "Transaction Summary", "================================================================================",
│ "Install 1 Package", "", "Total size: 89 k", "Installed size: 189 k", "Downloading Packages:", "Running transaction check", "Transaction check succeeded.", "Running
│ transaction test"]}
│ fatal: [3.83.34.169]: FAILED! => {"changed": true, "cmd": "sudo yum localinstall Postgres12/RHEL8/drpm-* -y --nogpgcheck \nsudo yum localinstall
│ Postgres12/RHEL8/createrepo_c-libs* -y --nogpgcheck\nsudo yum localinstall Postgres12/RHEL8/createrepo_c-0* -y --nogpgcheck\n", "delta": "0:00:38.637174", "end":
│ "2022-05-31 22:03:15.089383", "msg": "non-zero return code", "rc": 1, "start": "2022-05-31 22:02:36.452209", "stderr": "Error: Transaction test error:\n package
│ drpm-0.4.1-3.el8.x86_64 is already installed\n\nError: Transaction test error:\n package createrepo_c-libs-0.17.7-3.el8.x86_64 is already installed\n\nError:
│ Transaction test error:\n package createrepo_c-0.17.7-3.el8.x86_64 is already installed", "stderr_lines": ["Error: Transaction test error:", " package
│ drpm-0.4.1-3.el8.x86_64 is already installed", "", "Error: Transaction test error:", " package createrepo_c-libs-0.17.7-3.el8.x86_64 is already installed", "",
│ "Error: Transaction test error:", " package createrepo_c-0.17.7-3.el8.x86_64 is already installed"], "stdout": "Updating Subscription Management
│ repositories.\nUnable to read consumer identity\n\nThis system is not registered to Red Hat Subscription Management. You can use subscription-manager to
│ register.\n\nWaiting for process with pid 5943 to finish.\nLast metadata expiration check: 0:00:10 ago on Tue 31 May 2022 10:02:51 PM UTC.\nDependencies
│ resolved.\n================================================================================\n Package Architecture Version Repository
│ Size\n================================================================================\nInstalling:\n drpm x86_64 0.4.1-3.el8 @commandline
│ 68 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size: 68 k\nInstalled
│ size: 136 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nUpdating Subscription Management
│ repositories.\nUnable to read consumer identity\n\nThis system is not registered to Red Hat Subscription Management. You can use subscription-manager to
│ register.\n\nWaiting for process with pid 5965 to finish.\nLast metadata expiration check: 0:00:16 ago on Tue 31 May 2022 10:02:51 PM UTC.\nDependencies
│ resolved.\n================================================================================\n Package Arch Version Repository
│ Size\n================================================================================\nInstalling:\n createrepo_c-libs x86_64 0.17.7-3.el8
│ @commandline 118 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size:
│ 118 k\nInstalled size: 258 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nUpdating Subscription
│ Management repositories.\nUnable to read consumer identity\n\nThis system is not registered to Red Hat Subscription Management. You can use subscription-manager to
│ register.\n\nLast metadata expiration check: 0:00:20 ago on Tue 31 May 2022 10:02:51 PM UTC.\nDependencies
│ resolved.\n================================================================================\n Package Architecture Version Repository
│ Size\n================================================================================\nInstalling:\n createrepo_c x86_64 0.17.7-3.el8
│ @commandline 89 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size:
│ 89 k\nInstalled size: 189 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test", "stdout_lines": ["Updating
│ Subscription Management repositories.", "Unable to read consumer identity", "", "This system is not registered to Red Hat Subscription Management. You can use
│ subscription-manager to register.", "", "Waiting for process with pid 5943 to finish.", "Last metadata expiration check: 0:00:10 ago on Tue 31 May 2022 10:02:51 PM
│ UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version
│ Repository Size", "================================================================================", "Installing:", " drpm x86_64
│ 0.4.1-3.el8 @commandline 68 k", "", "Transaction Summary", "================================================================================",
│ "Install 1 Package", "", "Total size: 68 k", "Installed size: 136 k", "Downloading Packages:", "Running transaction check", "Transaction check succeeded.", "Running
│ transaction test", "Updating Subscription Management repositories.", "Unable to read consumer identity", "", "This system is not registered to Red Hat Subscription
│ Management. You can use subscription-manager to register.", "", "Waiting for process with pid 5965 to finish.", "Last metadata expiration check: 0:00:16 ago on Tue
│ 31 May 2022 10:02:51 PM UTC.", "Dependencies resolved.", "================================================================================", " Package
│ Arch Version Repository Size", "================================================================================", "Installing:", "
│ createrepo_c-libs x86_64 0.17.7-3.el8 @commandline 118 k", "", "Transaction Summary",
│ "================================================================================", "Install 1 Package", "", "Total size: 118 k", "Installed size: 258 k",
│ "Downloading Packages:", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Updating Subscription Management repositories.",
│ "Unable to read consumer identity", "", "This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.", "", "Last
│ metadata expiration check: 0:00:20 ago on Tue 31 May 2022 10:02:51 PM UTC.", "Dependencies resolved.",
│ "================================================================================", " Package Architecture Version Repository
│ Size", "================================================================================", "Installing:", " createrepo_c x86_64 0.17.7-3.el8
│ @commandline 89 k", "", "Transaction Summary", "================================================================================", "Install 1 Package", "",
│ "Total size: 89 k", "Installed size: 189 k", "Downloading Packages:", "Running transaction check", "Transaction check succeeded.", "Running transaction test"]}
│
│ PLAY RECAP *********************************************************************
│ 3.83.34.169 : ok=5 changed=2 unreachable=0 failed=1 skipped=6 rescued=0 ignored=0
│ 3.83.94.204 : ok=5 changed=2 unreachable=0 failed=1 skipped=6 rescued=0 ignored=0
│
│
╵
I set the logging to terraform.log and ran the apply with TF_LOG=DEBUG OS_DEBUG=true terraform apply But, you can see that when I tail the log, it cuts out at a sink error, and then doesn't record anything for about 10 minutes while the deployment runs, then picks back up with a few errors at the end:
2022-05-31T22:02:03.798Z [DEBUG] opening new ssh session
2022-05-31T22:02:03.819Z [DEBUG] starting remote command: /tmp/terraform_882212444.sh
2022-05-31T22:02:04.346Z [DEBUG] remote command exited with '0': /tmp/terraform_882212444.sh
2022-05-31T22:02:04.346Z [DEBUG] opening new ssh session
2022-05-31T22:02:04.355Z [DEBUG] Starting remote scp process: 'scp' -vt /tmp
2022-05-31T22:02:04.399Z [DEBUG] Started SCP session, beginning transfers...
2022-05-31T22:02:04.399Z [DEBUG] Copying input data into temporary file so we can read the length
2022-05-31T22:02:04.400Z [DEBUG] Beginning file upload...
2022-05-31T22:02:04.653Z [DEBUG] SCP session complete, closing stdin pipe.
2022-05-31T22:02:04.653Z [DEBUG] Waiting for SSH session to complete.
2022-05-31T22:02:04.665Z [ERROR] scp stderr: "Sink: C0644 0 terraform_882212444.sh\n"
2022-05-31T22:03:15.271Z [WARN] Errors while provisioning null_resource.ExecuteAnsible[0] with "l ocal-exec", so aborting
2022-05-31T22:03:15.275Z [ERROR] vertex "null_resource.ExecuteAnsible[0]" error: local-exec provis ioner error
2022-05-31T22:09:59.841Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error : code = Unavailable desc = transport is closing"
2022-05-31T22:09:59.842Z [DEBUG] provider: plugin process exited: path=.terraform/providers/regis try.terraform.io/hashicorp/null/3.1.1/linux_amd64/terraform-provider-null_v3.1.1_x5 pid=8933
2022-05-31T22:09:59.842Z [DEBUG] provider: plugin exited
I don't know how to resolve this sink error, I don't know why my logging is cutting out for 10 minutes. I also can't figure out why the job is marked as failed even though I see 0 failures in the summary, and the app has been deployed. Any help would be so much appreciated. been banging my head against the wall for a week on this.
The "Sink: C0644" errors and the "provider.stdio: received EOF, stopping recv loop" errors were just red herrings. The actual problem was that the way I had my null_resource set, it was trying to execute each playbook on each node twice. Here's the corrected: