Using testcotainers in a container for integration tests in Atlassian Bitbucket pipeline

Question

I'm using a .NET 7 docker container to run some tests in an Atlassian Bitbucket pipeline. The integration tests in my solution rely on testcontainers library to set-up a mssql container where my application can connect to.

I am able to run these integration tests from a container on my local machine. When I run the container I can see it interacting with my Docker for Windows app and setting up new container dependecies defined in the tests. However, I am having issues replicating this setup in a bitbucket pipeline step. When I run it on bitbucket I am getting an error when my integration tests get to the step where testcontainers attempts to setup its dependencies in the test:

Stack trace error:

    tests    |   Failed DockerTestContainers.IntegrationTets.CustomerController.CreateCustomerControllerTests.Create_CreateCustomer_WhenDataIsValid [1 ms]
    tests    |   Error Message:
    tests    |    System.AggregateException : One or more errors occurred. (Docker is either not running or misconfigured. Please ensure that Docker is running and that the endpoint is properly configured. You can customize your configuration using either the environment variables or the ~/.testcontainers.properties file. For more information, visit:
    tests    | https://dotnet.testcontainers.org/custom_configuration/ (Parameter 'DockerEndpointAuthConfig')) (The following constructor parameters did not have matching fixture data: CustomWebApplicationFactory`1 factory)
    tests    | ---- System.ArgumentException : Docker is either not running or misconfigured. Please ensure that Docker is running and that the endpoint is properly configured. You can customize your configuration using either the environment variables or the ~/.testcontainers.properties file. For more information, visit:
    tests    | https://dotnet.testcontainers.org/custom_configuration/ (Parameter 'DockerEndpointAuthConfig')
    tests    | ---- The following constructor parameters did not have matching fixture data: CustomWebApplicationFactory`1 factory
    tests    |   Stack Trace:
    tests    |   
    tests    | ----- Inner Stack Trace #1 (System.ArgumentException) -----
    tests    |    at DotNet.Testcontainers.Guard.ThrowIf[TType](ArgumentInfo`1& argument, Func`2 condition, Func`2 ifClause)
    tests    |    at DotNet.Testcontainers.Builders.AbstractBuilder`4.Validate()
    tests    |    at DotNet.Testcontainers.Builders.ContainerBuilder`3.Validate()
    tests    |    at Testcontainers.MsSql.MsSqlBuilder.Validate()
    tests    |    at Testcontainers.MsSql.MsSqlBuilder.Build()
    tests    |    at DockerTestContainers.IntegrationTets.CustomWebApplicationFactory`1..ctor() in /src/DockerTestContainers.IntegrationTets/CustomWebApplicationFactory.cs:line 18
    tests    |    at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    tests    |    at System.Reflection.ConstructorInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
    tests    | ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
    tests    | 
    tests    | Results File: /src/DockerTestContainers.IntegrationTets/TestResults/_57b440a6d993_2024-03-05_22_59_22.trx
    tests    | 
    tests    | Failed!  - Failed:     4, Passed:     0, Skipped:     0, Total:     4, Duration: 9 ms - DockerTestContainers.IntegrationTets.dll (net7.0)

This is the bitbucket-pipelines.yml I have setup:

    image: atlassian/default-image:3
        
    pipelines:
      default:
        - parallel:
          - step:
              name: 'Build and Test'
              services:
                - docker
              script:
                - cd "DockerTestContainers"
                - ls
                - export TESTCONTAINERS_RYUK_DISABLED=true
                - docker-compose -f docker-compose-api-tests.yaml build
                - docker-compose -f docker-compose-api-tests.yaml up --exit-code-from tests

This is my docker-compose-api-test.yml content that my bitbucket pipeline build step is calling:

    version: '3.7'
    services:
        tests:
            command: --filter Category=MyIntegrationTests --logger:trx
            container_name: tests
            environment:
                - AWS_DEFAULT_REGION=us-east-1
                - AWS_ACCESS_KEY_ID=xxx
                - AWS_SECRET_ACCESS_KEY=xxx
                - TESTCONTAINERS_HOST_OVERRIDE=host.docker.internal
            build:
                context: .
                target: integrationtests
                dockerfile: DockerfileTest
            volumes:
            - /var/run/docker.sock:/var/run/docker.sock

Has anyone had experienced getting testcontainer integration tests running successfully in bitbucket pipelines? Not sure I understand the problem here if bitbucket step is using the docker servicse already.

I have seen some comments recommending to use docker:dind (docker in docker) image for the pipeline setup. However, I get an error when I attempt to use the specific "docker:dind" image name in my step.

I have also tried with and without TESTCONTAINERS_RYUK_DISABLED=true setting. I have seen a few posts that have recommended to disable this setting in similarly described scenarios.