What is the correct hostname for pgloader when using this tool in docker-in-docker image?

Question

I'm trying to run two databases (msql and postgres) inside a container with docker installed. Unfortunately, when using pgloader, there is an error connecting to the databases. Here are the steps I take:

1. i am pulling dind image: docker pull docker:dind
2. running container from dind image: docker run --privileged --name dind-container --hostname lch -d docker:dind

• now i enter commands below inside container "dind-container"

3. installing pgloader to container: apk add --no-cache pgloader
4. get images of msql db and postgres: docker pull -q mysql:8 and docker pull -q postgres:16
5. create network: docker network create my-network || true
6. run mysql container:

docker run -p 3306:3306 -d --network my-network --name mysql-db --hostname mysql -e MYSQL_ROOT_PASSWORD=mysql -e MYSQL_ROOT_HOST=root -e MYSQL_DATABASE=my_db -e MYSQL_USER=my_user -e MYSQL_PASSWORD=my_password mysql:8 --default-authentication-plugin=mysql_native_password

7. run postgres database:

docker run -d --network my-network --name postgres-db --hostname postgres -e POSTGRES_DB=my_db -e POSTGRES_USER=my_user -e POSTGRES_PASSWORD=my_password postgres:16

• at this moment it looks like this inside container:

8. waiting for setting up databases
9. run pgloader command:

pgloader --with "batch concurrency = 1" mysql://my_user:my_password@lch/my_db postgresql://my_user:my_password@$lch/my_db

My question is here: what to enter for 'lch'. I have tried various things, including 'localhost', but there is always an error. I tried logging ip like this and pasting as host: ip route|awk '{print $3; exit}'.

Error:

I will add that I managed to do this locally on my machine, then I simply used 'localhost'. However, now I have to use dind image and do it inside container and this is the problem. I also know that databases are not populated with tables, but that is not a problem here.

Answer 1

If you run the database with a correct docker run -p option to publish its ports

docker run -p 5432:5432 ... postgres:16
#          ^^^^^^^^^^^^

then, from a non-container process inside the DinD container, you should be able to access localhost:5432 to reach the database. As with other -p options, your non-container tool needs to connect to the first -p port, and the second -p port must match the standard port the process inside the container is using.

If you want to access this port further from the actual host system, you also need to start the DinD container with a similar -p option.

Answer 2

Ok, thanks for your help, I managed to figure it out. Additionally, I had some omissions and typo in commands above. As @David Maze wrote, port was missing in this command docker run -d --network my-network --name postgres-db --hostname postgres -e POSTGRES_DB=my_db -e POSTGRES_USER=my_user -e POSTGRES_PASSWORD=my_password postgres:16, and I have '$' unnecessarily here: ...@$lch/my_db.

Below I put two working solutions:

1. with ports, without network, with pgloader downloaded,
2. without ports, with network, with downloaded container inside which pgloader is installed.

Solution 1:

docker pull docker:dind 
docker run --privileged --name dind-container --hostname lch -d docker:dind

inside "dind-container"

apk add --no-cache pgloader  
docker pull -q mysql:8 
docker pull -q postgres:16  
docker run -p 3306:3306 -d --name mysql-db --hostname mysql -e MYSQL_ROOT_PASSWORD=mysql -e MYSQL_ROOT_HOST=root -e MYSQL_DATABASE=my_db -e MYSQL_USER=my_user -e MYSQL_PASSWORD=my_password mysql:8 --default-authentication-plugin=mysql_native_password
docker run -d -p 5432:5432 --name postgres-db --hostname postgres -e POSTGRES_DB=my_db -e POSTGRES_USER=my_user -e POSTGRES_PASSWORD=my_password postgres:16 
pgloader --with "batch concurrency = 1" mysql://my_user:my_password@localhost/my_db postgresql://my_user:my_password@localhost/my_db

or

pgloader --with "batch concurrency = 1" mysql://my_user:my_password@lch/my_db postgresql://my_user:my_password@lch/my_db

^ both commands work

Solution 2:

docker pull docker:dind  
docker run --privileged --name dind-container --hostname lch -d docker:dind

inside "dind-container"

apk add --no-cache pgloader 
docker pull -q mysql:8 
docker pull -q postgres:16  
docker pull dimitri/pgloader  
docker network create my-network || true  
docker run -d --network my-network --name mysql-db --hostname mysql -e MYSQL_ROOT_PASSWORD=mysql -e MYSQL_ROOT_HOST=root -e MYSQL_DATABASE=my_db -e MYSQL_USER=my_user -e MYSQL_PASSWORD=my_password mysql:8 --default-authentication-plugin=mysql_native_password  
docker run -d --network my-network --name postgres-db --hostname postgres -e POSTGRES_DB=my_db -e POSTGRES_USER=my_user -e POSTGRES_PASSWORD=my_password postgres:16  
docker run --network my-network dimitri/pgloader pgloader --with "batch concurrency = 1" mysql://my_user:my_password@mysql/my_db postgresql://my_user:my_password@postgres/my_db