Cloudwatch Custom Events SQS fails to work

Question

I am using terraform to create the queues while also creating the Cloudwatch event rules and setting one of the queues as the target for the rules.

In summary I have a single queue that is the target for 3 separate cloudwatch events. The problem is that even though the cloudwatch event rules are identical, only one of them work when created through terraform, the others end up as failed invocations in the console with no log or any sort of debuggable information. If the custom events are created from the aws console, all works well.

The creation of the queue in terraform

resource "aws_sqs_queue" "queue_cron" {
  name                       = "cron"
  visibility_timeout_seconds = 300 # 5 minutes
  delay_seconds              = 0
  message_retention_seconds  = 1800 # 30 minutes
  receive_wait_time_seconds  = 20
}

The only working block

resource "aws_cloudwatch_event_rule" "eve_vendors_bot_sync" {
  name                = "vendors-bot-sync"
  schedule_expression = "rate(1 minute)"
  description         = "Notify cron queue for vendors bot sync"
  is_enabled          = true
}

resource "aws_cloudwatch_event_target" "sqs_cron_vendors_bot_sync" {
  rule      = aws_cloudwatch_event_rule.eve_vendors_bot_sync.name
  arn       = var.queue_cron_arn
  target_id = "sqsCronVendorBotSync"

  input_transformer {
    input_template = <<EOF
{
   "messageType":"cron",
   "cronType":"vendors-bot-sync"
}
EOF
  }
}

Doesn't work even though it's identical in structure to the one above.

resource "aws_cloudwatch_event_rule" "eve_restos_sync" {
  name                = "restos-sync"
  schedule_expression = "rate(1 minute)"
  description         = "Notify cron queue for restos sync"
  is_enabled          = true
}

resource "aws_cloudwatch_event_target" "sqs_cron_restos_sync" {
  rule      = aws_cloudwatch_event_rule.eve_restos_sync.name
  arn       = var.queue_cron_arn
  target_id = "sqsCronRestosSync"

  input_transformer {
    input_template = <<EOF
{
   "messageType":"cron",
   "cronType":"restaurant-hours-open-close-management"
}
EOF
  }
}

Similar to the one above, does not work

resource "aws_cloudwatch_event_rule" "eve_vendors_orders_sync" {
  name                = "vendors-orders-sync"
  schedule_expression = "rate(1 minute)"
  description         = "Notify cron queue for vendors orders sync"
  is_enabled          = true
}
resource "aws_cloudwatch_event_target" "target_cron_vendors_sync" {
  rule      = aws_cloudwatch_event_rule.eve_vendors_orders_sync.name
  arn       = var.queue_cron_arn
  target_id = "sqsCronVendorsOrderSync"

  input_transformer {
    input_template = <<EOF
{
   "messageType":"cron",
   "cronType":"vendors-orders-sync"
}
EOF
  }
}

Answer

The missing piece in the puzzle as rightfully pointed out by @Marchin was indeed the policy that was preventing cloudwatch from sending a message to SQS. Here is the updated config that got it working.

1. Create the queue
2. Create a policy that will allow cloudwatch the ability to sendMessages to the queue
3. Attach the policy to the queue

resource "aws_sqs_queue" "queue_cron" {
  name                       = "cron"
  visibility_timeout_seconds = 300 # 5 minutes
  delay_seconds              = 0
  message_retention_seconds  = 1800 # 30 minutes
  receive_wait_time_seconds  = 20
}

data "aws_iam_policy_document" "policy_sqs" {
  statement {

    sid    = "AWSEvents_"
    effect = "Allow"
    actions = [
      "sqs:SendMessage",
    ]

    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }

    resources = [aws_sqs_queue.queue_cron.arn]

  }
}

resource "aws_sqs_queue_policy" "cron_sqs_policy" {

  queue_url = aws_sqs_queue.queue_cron.id
  policy    = data.aws_iam_policy_document.policy_sqs.json

}

Answer 1

I think your permissions on SQS queue are missing or are incorrect. Assuming that you are creating your queue_cron in terraform (not shown in the question), the queue and its policy allowing CW Events sending messages to it would be:

data "aws_caller_identity" "current" {}

data "aws_region" "current" {}

resource "aws_sqs_queue" "queue_cron" {
  name   = "queue_cron"
}

resource "aws_sqs_queue_policy" "test" {

  queue_url = aws_sqs_queue.queue_cron.id

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": {
        "AWS": "${data.aws_caller_identity.current.account_id}"
      },
      "Action": "sqs:*",
      "Resource": "${aws_sqs_queue.queue_cron.arn}"
    },
    {
      "Sid": "AWSEvents_",
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sqs:SendMessage",
      "Resource": "${aws_sqs_queue.queue_cron.arn}",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:events:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:rule/*"
        }
      }
    }  
  ]
}
POLICY
}