I'm struggling with cyclic dependency issue in terraform apply stage when using depends_on on a module.
The error I got in apply is:
* Cycle: aws_appautoscaling_policy.queue_depth_based_scale_out_policy, module.my_module.aws_ecs_task_definition.task_definition (destroy), aws_appautoscaling_policy.queue_depth_based_scale_in_policy
The plan stage looked exactly fine and there are no errors in the plan stage.
I have tried to identify the cycle in graph using below command,
terraform graph -draw-cycles -module-depth=0 -type=plan | dot -Tsvg > graph-plan.svg
There is no cycle shown in plan graph.
Then, tried to identify cycle in apply using,
terraform graph -draw-cycles -module-depth=0 -type=apply | dot -Tsvg > graph-apply.svg
Sadly, this commands is unable to show cycle in graph.
Fortunately, I am able to see the cycle in apply stage graph using these below commands,
terraform plan -out tfplan
terraform graph -draw-cycles -module-depth=0 tfplan | dot -Tsvg > graph-apply.svg
The cycle in my graph looks like this,
Although, I am still unable to make out for the reason for this cycle in graph.
Moreover, it seems issue is specifically with adding depends_on on a module.
Since I already have few more aws_appautoscaling_policy in my module which do depends on aws_appautoscaling_target which depends on aws_ecs_service and hence eventually depends on aws_ecs_task_definition but apply for this works fine.
There are some aws_appautoscaling_policy which are specifically related to a particular app hence I am adding them separately (and not as part of module), but since auto scaling policy can only be added once service is registered as scalable target, hence I am adding depends_on on module, since aws_appautoscaling_target is defined in the module.
Here is my code snippet for module,
resource "aws_ecs_task_definition" "task_definition" {
family = "${var.service_name}"
container_definitions = "${var.container_definitions}"
task_role_arn = "${aws_iam_role.task_role.arn}"
lifecycle {
create_before_destroy = true
}
}
resource "aws_ecs_service" "service" {
name = "${var.service_name}"
cluster = "${data.aws_ecs_cluster.ecs_cluster.arn}"
task_definition = "${aws_ecs_task_definition.task_definition.arn}"
deployment_minimum_healthy_percent = 50
deployment_maximum_percent = 100
lifecycle {
ignore_changes = ["desired_count"]
}
}
resource "aws_appautoscaling_target" "ecs_target" {
max_capacity = "${var.max_scalabe_capacity}"
min_capacity = "${var.min_scalabe_capacity}"
resource_id = "service/${var.ecs_cluster_name}/${aws_ecs_service.service.name}"
scalable_dimension = "ecs:service:DesiredCount"
service_namespace = "ecs"
}
resource "aws_appautoscaling_policy" "cpu_based_scale_in_policy" {
name = "${var.service_name}-${var.env}-cpu-based-scale-in-policy"
policy_type = "StepScaling"
resource_id = "service/${var.ecs_cluster_name}/${var.service_name}"
scalable_dimension = "ecs:service:DesiredCount"
service_namespace = "ecs"
step_scaling_policy_configuration {
adjustment_type = "ChangeInCapacity"
cooldown = "${var.scale_in_cooldown_period}"
metric_aggregation_type = "Average"
step_adjustment {
metric_interval_upper_bound = "${var.scale_in_step_adjustment_upper_bound}"
scaling_adjustment = "${var.scale_in_step_adjustment_scaling_adjustment}"
}
}
depends_on = ["aws_appautoscaling_target.ecs_target"]
}
And here is the usage of module,
module "my_module" {
source = "GIT_URL_FOR_MODULE"
VARIABLES_AS_NEEDED_BY_MODULE
}
resource "aws_appautoscaling_policy" "queue_depth_based_scale_in_policy" {
name = "${local.service_name}-${local.env}-queue-scale-in-policy-new"
policy_type = "StepScaling"
resource_id = "service/${local.ecs_cluster_name}/${local.service_name}"
scalable_dimension = "ecs:service:DesiredCount"
service_namespace = "ecs"
step_scaling_policy_configuration {
adjustment_type = "ChangeInCapacity"
cooldown = "${local.queue_scale_in_cooldown_period}"
metric_aggregation_type = "Average"
step_adjustment {
metric_interval_upper_bound = "${local.queue_scale_in_step_adjustment_upper_bound}"
scaling_adjustment = "${local.queue_scale_in_step_adjustment_scaling_adjustment}"
}
}
depends_on = ["module.my_module"]
}
Steps followed in pipeline are:
terraform get -update=true
terraform init
terraform taint -allow-missing -module=${MODULE_NAME} aws_ecs_task_definition.task_definition
terraform plan -out tfplan -input=false
terraform apply -input=false tfplan
Would be happy to learn the reason behind this cycle?
Another point to highlight is that terraform apply is successful when we just destroy everything and recreate from scratch. The cycle is only observed when I taint my task-definition and have some updates in my scaling policies which are placed outside of module.
Note: In my pipeline, I do taint the previous task definition, to ensure services are started with new task definition instantaneously, otherwise task(s) will not be immediately rolled out with new task definition.
I have managed to get rid of cyclic dependency. Here is the approach used,
Rather than having a dependency on the entire module, I have added output for
aws_appautoscaling_targetin the module. And then I am just using this output in the scaling policy to ensure there is an implicit dependency created.Here is the sample code,
Module
Note the output code block added inside the module. And here is the usage of module, where output of module is consumed in scaling policy name.
Although, I am still unable to figure out why the cycle was there in the first place.