Use an existing EBS volume with the Docker REX-Ray plugin on ECS

Question

I'm using Terraform to create an ECS Cluster, a task definition and a service.

For task definition, the piece of code looks like this:

resource "aws_ecs_task_definition" "postgres" {
  container_definitions    = ...
  family                   = "Postgres"
  requires_compatibilities = ["EC2"]
  network_mode             = "bridge"

  volume {
    name = "PreCreatedEBSVolume"

    docker_volume_configuration {
      scope         = "task"
      autoprovision = false
      driver        = "rexray/ebs"
    }
  }
}

I specifically set autoprovision to false to make sure a new volume is not created. But still, when the service runs, a new volume is created with 16GiB (which I assume is the default size) and used instead of using the "PreCreatedEBSVolume".

I check the docker volume ls in the instance and sure enough, I see the EBS volumes listed. And if I run a new instance with this volume mounted, it's mounting as expected.

Is there a way I can make REX-Ray use existing block storage?

Answer 1

I found the issue.

The problem is the "scope" property. The docs had the following to say:

The scope for the Docker volume that determines its lifecycle. Docker volumes that are scoped to a task are automatically provisioned when the task starts and destroyed when the task stops. Docker volumes that are scoped as shared persist after the task stops.

So, I just had to change it to "shared" from "task" to make it use the existing EBS volume.

Answer 2

Problem with rexray/ebs is that it selects existing EBS volume only when it is in same AZ as instance. So in our case we needed to pin specific instances to specific AZ in order to use this driver properly.

Our setup was 6 instances in 3 different AZ for better availability and every instance had its own EBS, but after instance restart it went to random AZ and it was 2 out of 3 it failed to find the volume. Error message in syslogs was not very informative.

Other then that plugin does what it should.

Answer 3

Following is the task definition example that answers the question as well as gives extra details if you want to know how to create a gp3 volume with IOPS and encryption specified:

resource "aws_ecs_task_definition" "postgres" {
  container_definitions    = ...
  family                   = "Postgres"
  requires_compatibilities = ["EC2"]
  network_mode             = "bridge"

  volume {
    name = "PreCreatedEBSVolume"

    docker_volume_configuration {
      scope         = "shared"
      autoprovision = false
      driver        = "rexray/ebs"
      driver_opts = {
        volumetype    = "gp3"
        size          = 40
        iops          = 3000
        encrypted     = true
        encryptionkey = "arn:aws:kms:us-east-1:111111111111:key/11111111-1111-1111-1111-11111111"
      }
    }
  }
}