Activate conda environment in docker and using image in WDL

Question

I am developing dockerfile where i need to activate conda environment(working fine)

Dockerfile:

FROM continuumio/miniconda3

WORKDIR /app

# Create the environment:
COPY environment.yml .
RUN conda env create -f environment.yml

# Make RUN commands use the new environment:
SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]


# The code to run when container is started:
COPY test.py .
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "myenv"]

environment.yml

name: myenv
channels:
  - conda-forge
dependencies:
  - python=3.8
  - flask
  - numpy

test.py

import numpy as np
 
# Creating a rank 1 Array
arr = np.array([1, 2, 3])
print("Array with Rank 1: \n",arr)

Building and pushing docker image:

docker build -t docker.io/myaccount/condaset:latest .
docker push docker.io/myaccount/condaset:latest

I tested the docker image which seems to be working fine.

docker run docker.io/myaccount/condaset python test.py 
Array with Rank 1: 
 [1 2 3]

but when i use the same docker image in WDL it gives error.

test.wdl

version 1.0
task my_run
{
    input
    {
      String somevar
    }

    parameter_meta {
    }

    command {
       set -exo pipefail
       python /app/test.py 
    }
    
    output { 
       File out = ''
    }
    
    runtime {
       docker: "docker.io/myaccount/condaset:latest"
    }
}
workflow my_wokflow
{
    input
    {
       String somevar
    }

    call my_run
    {
      input: somevar=somevar
    }
}

input.json

{
  "my_wokflow.somevar": "hello_world"
}

Error while running WDL:

java -jar ~/bin/cromwell-58.jar run test.wdl -i input.json

 [First 3000 bytes]:+ python /app/test.py
Traceback (most recent call last):
  File "/app/test.py", line 1, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'

It seems conda environment become deactivated before calling dockerimage in WDL. How can i fix such issue?

Answer 1

When cromwell runs a container, it overrides the ENTRYPOINT script, so your conda environment does not get activated when the WDL is run.

There are a a couple of different ways you can get the behavior you want.

1. Add your conda environment to the your PATH in the Dockerfile:

   ENV PATH="/opt/conda/envs/myenv/bin:$PATH"
   

2. In your WDL, supply an absolute path to python:

   /opt/conda/envs/myenv/bin/python /app/test.py 
   

Additionally, in your WDL, the my_run output stanza will result in an error. You need to give a non-empty file name for out:

    command {
       set -exo pipefail
       /opt/conda/envs/myenv/bin/python /app/test.py  > out.txt
    }   
    
    output { 
       File out = 'out.txt'
    }