airflow using DockerOperator and did (Docker in Docker) intependend from host docker service

52 views Asked by At

TL:DR; Use Docker-in-Docker in airflow, not using the host docker service

My airflow is running in as docker compose. I want to add Docker in Docker as a service so that the DockerOperator is completely independent from the host docker (therefore not mapping /var/run/docker.sock:/var/run/docker.sock into airflow)

My docker compose (left out most of the irrelevant stuff):

version: '3'
x-airflow-common:
    &airflow-common
    build: services/airflow/.
    environment:
        &airflow-common-env
        ...
    volumes:
        ...
        # NOT exposing the host docker service
        # - /var/run/docker.sock:/var/run/docker.sock

services:
    dind:
        image: docker:dind
        privileged: true  # Use privileged mode to run Docker-in-Docker
        volumes:
        - /var/lib/docker
        networks:
        - airflow

    airflow-webserver: 
        <<: *airflow-common 
        command: webserver
        ports:
            - 127.0.0.1:8080:8080 
        networks:
            - airflow

    airflow-scheduler:
        <<: *airflow-common 
        command: scheduler

    airflow-worker:
        <<: *airflow-common 
        command: celery worker -c 8

    airflow-init: 
        <<: *airflow-common 
        command: version
        environment:
            <<: *airflow-common-env
            _AIRFLOW_DB_MIGRATE: 'true'
            _AIRFLOW_WWW_USER_CREATE: 'true'
            _AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow} 
            _AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
        networks:
            - airflow

networks:
  airflow:

My DockerOperator:

docker1 = DockerOperator(
        docker_url='tcp://dind:2375',
        image="python:3.9.7-slim-buster",
        command="python --version",
        auto_remove=True,
        dag=dag,
    )

Getting the error inside the taks:
docker.errors.DockerException: Error while fetching server API version: HTTPConnectionPool(host='dind', port=2375): Max retries exceeded with url: /version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0a30e60730>: Failed to establish a new connection: [Errno 111] Connection refused'))

Also, from the airflow-worker i can not reach did:
DOCKER_HOST=tcp://dind:2375 docker info connection refused

0

There are 0 answers