Running Dagster on Docker: "ImportError: cannot import name 'Literal' from 'typing' (/usr/local/lib/python3.7/typing.py)"

Question

I'm trying to set up dagster to run on Docker, eventually on Docker Swarm + Portainer, however while setting this up I'm getting this error:

ImportError: cannot import name 'Literal' from 'typing' (/usr/local/lib/python3.7/typing.py)

Full stacktrace here:

Traceback (most recent call last):
2023-10-20 13:35:47   File "/usr/local/bin/dagster-daemon", line 5, in <module>
2023-10-20 13:35:47     from dagster.daemon.cli import main
2023-10-20 13:35:47   File "/usr/local/lib/python3.7/site-packages/dagster/__init__.py", line 480, in <module>
2023-10-20 13:35:47     from dagster._core.pipes.client import (
2023-10-20 13:35:47   File "/usr/local/lib/python3.7/site-packages/dagster/_core/pipes/client.py", line 5, in <module>
2023-10-20 13:35:47     from dagster_pipes import (
2023-10-20 13:35:47   File "/usr/local/lib/python3.7/site-packages/dagster_pipes/__init__.py", line 15, in <module>
2023-10-20 13:35:47     from typing import (
2023-10-20 13:35:47 ImportError: cannot import name 'Literal' from 'typing' (/usr/local/lib/python3.7/typing.py)
2023-10-20 13:53:50 Traceback (most recent call last):
2023-10-20 13:53:50   File "/usr/local/bin/dagster-daemon", line 5, in <module>
2023-10-20 13:53:50     from dagster.daemon.cli import main
2023-10-20 13:53:50   File "/usr/local/lib/python3.7/site-packages/dagster/__init__.py", line 480, in <module>
2023-10-20 13:53:50     from dagster._core.pipes.client import (
2023-10-20 13:53:50   File "/usr/local/lib/python3.7/site-packages/dagster/_core/pipes/client.py", line 5, in <module>
2023-10-20 13:53:50     from dagster_pipes import (
2023-10-20 13:53:50   File "/usr/local/lib/python3.7/site-packages/dagster_pipes/__init__.py", line 15, in <module>
2023-10-20 13:53:50     from typing import (
2023-10-20 13:53:50 ImportError: cannot import name 'Literal' from 'typing' (/usr/local/lib/python3.7/typing.py)

My Dockerfile looks like this:

FROM python:3.11-slim

WORKDIR /usr/src/app
ENV DAGSTER_HOME=/usr/src/app


RUN pip install dagster dagster-webserver dagit dagster-postgres SQLAlchemy==1.4.49 pandas pyarrow

# # Copy our code and workspace to /usr/src/app
COPY dagster.yaml
# COPY etl ./etl
COPY pyproject.toml .
COPY setup.cfg .
COPY setup.py .

EXPOSE 3000

CMD ["dagster-webserver", "-h", "0.0.0.0", "-p", "3000"]

This is the Dagster yaml file:

run_storage:
  module: dagster_postgres.run_storage
  class: PostgresRunStorage
  config:
    postgres_db:
      username: postgres
      password: demopass
      hostname: dagster-postgres
      db_name: postgres
      port: 5432

event_log_storage:
  module: dagster_postgres.event_log
  class: PostgresEventLogStorage
  config:
    postgres_db:
      username: postgres
      password: demopass
      hostname: dagster-postgres
      db_name: postgres
      port: 5432

schedule_storage:
  module: dagster_postgres.schedule_storage
  class: PostgresScheduleStorage
  config:
    postgres_db:
      username: postgres
      password: demopass
      hostname: dagster-postgres
      db_name: postgres
      port: 5432

run_coordinator:
  module: dagster.core.run_coordinator
  class: QueuedRunCoordinator
  config:
    max_concurrent_runs: 5
    
telemetry:
  enabled: false
nux:
  enabled: false

And the Docker compose yaml:

services:

  dagster-dagit:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 3000:3000

  dagster-daemon:
    build:
      context: .
      dockerfile: Dockerfile
    command: "dagster-daemon run"
    environment:
      - PGPASS=${PGPASS}
      - PGUID=${PGUID}

  dagster-postgres:
    image: postgres:13.3
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
    volumes:
      - dagster-postgres:/var/lib/postgresql/data
volumes:
  dagster-postgres:
    driver: local

I thought it might be a version incompatibility issue with pandas, however that seems fine. Also looked at a few similar threads suggesting that ENV variables might be missing, but mine are all set.

Any ideas?

Tried different versions to check compatibility and checked whether env variables are set properly. Expect to have all 3 containers to be up, however only the postgres one is running currently.

Answer 1

I can see the problem comes from Python 3.7 instead of the 3.11 of the image. What version gives you if you go inside the container and run the cmd which python?

docker exec -it CONTAINER_ID /bin/bash will do that trick.

I suggest downgrading to Python 3.10.10 and giving it a try.

At the same time, when you're inside the container in interactive mode, it would be worth checking if you have multiple pip versions, but everything points out to a path confusion between python/pip paths.