TL:DR; Use Docker-in-Docker in airflow, not using the host docker service
My airflow is running in as docker compose. I want to add Docker in Docker as a service so that the DockerOperator is completely independent from the host docker (therefore not mapping /var/run/docker.sock:/var/run/docker.sock into airflow)
My docker compose (left out most of the irrelevant stuff):
version: '3'
x-airflow-common:
&airflow-common
build: services/airflow/.
environment:
&airflow-common-env
...
volumes:
...
# NOT exposing the host docker service
# - /var/run/docker.sock:/var/run/docker.sock
services:
dind:
image: docker:dind
privileged: true # Use privileged mode to run Docker-in-Docker
volumes:
- /var/lib/docker
networks:
- airflow
airflow-webserver:
<<: *airflow-common
command: webserver
ports:
- 127.0.0.1:8080:8080
networks:
- airflow
airflow-scheduler:
<<: *airflow-common
command: scheduler
airflow-worker:
<<: *airflow-common
command: celery worker -c 8
airflow-init:
<<: *airflow-common
command: version
environment:
<<: *airflow-common-env
_AIRFLOW_DB_MIGRATE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
networks:
- airflow
networks:
airflow:
My DockerOperator:
docker1 = DockerOperator(
docker_url='tcp://dind:2375',
image="python:3.9.7-slim-buster",
command="python --version",
auto_remove=True,
dag=dag,
)
Getting the error inside the taks:
docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='dind', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0a30e60730>: Failed to establish a new connection: [Errno 111] Connection refused'))
Also, from the airflow-worker i can not reach did:
DOCKER_HOST=tcp://dind:2375 docker info
connection refused