Issue
We are attempting to run toil-cwl-runner with a default singularity image. We got this to work as we intended with cwltool, but toil-cwl-runner seems like it is ignoring the default image to use.
Attempt
What we do is we use the options "--singularity" and "--default-container" first with cwltool and then with toil-cwl-runner. To start, the versions of our OS and software are
- System OS: Ubuntu 22.04
- Singularity Image OS: 20.04 (Intentionally different to test if the container is being used)
- Singularity Version: 3.9.2
- toil Version: 5.12.0
- CWLTool Version: 3.1.20230601100705
- Python Version: 3.9.18
The .cwl workflow we call is intentionally very simple and intentionally split for illustrative purposes. It takes in a list of ints to add together, and a list of ints to multiply together both of which are given to separate python scripts to perform each action. Then it multiplies the two resulting numbers using the same multiply python script from before at which point it prints the result of all of these operations into a .txt file with 'The result is ' prepended to it. The file is called 'z_output.txt'. Additionally, it looks for the version of the current OS and prints this into a separate .txt file called 'z_version.txt'. We created this ultra simple workflow as a sample for our group and to verify that the singularity container was being used. To get this to run as intended, we defining a few shell variables for singularity to be able to find the image we will use, specifically:
export SINGULARITY_CACHEDIR=<Path/to/Singularity/.sif>
export SINGULARITY_TMPDIR=$SINGULARITY_CACHEDIR/tmp
export SINGULARITY_PULLDIR=$SINGULARITY_CACHEDIR/pull
export CWL_SINGULARITY_CACHE=$SINGULARITY_CACHEDIR
Then to show how it should work we call the workflow with cwltool using the command:
cwltool --singularity --default-container <imageName> workflow.cwl parameters.yaml
This results in the file, 'z_version.txt', containing the OS version of the singularity image we specified. If we try to do the same thing with toil-cwl-runner by calling
toil-cwl-runner --singularity --default-container <imageName> workflow.cwl parameters.yaml
What we expected is that this would print the same OS version into 'z_version.txt' as calling it with cwltool would. However, it prints the OS of the current system into the 'z_version.txt' file. One thing to note is that for some reason cwltool will replace the output files if they exist, but toil-cwl-runner does not replace the files if they already exists. We don't know exactly why, but we also aren't concerned with that at the moment. The fact that the OS printed into the file is that of the main system, and not the singularity images OS, leads us to believe that toil-cwl-runner is ignoring the "--default-container" option for each of the steps. The .cwl file that prints the version looks like this:
cwlVersion: v1.2
class: CommandLineTool
baseCommand: [cat, /etc/os-release]
stdout: z_version.txt
inputs: []
outputs:
output_file:
type: File
outputBinding:
glob: z_version.txt
To help with reproducibility, we also created a github repository with all the files needed and a brief explanation of how we call it, which is a tiny bit more detailed than this for those wanting to rerun our scripts.
Because the workflows finish in both cwltool and toil-cwl-runner, without printing out any warnings or errors, we did not include any logs as we were not sure which logs would be useful at this time. If any specific logs would be helpful, please do let us know and we will gladly add them when we can.
We are aware that we are not using any of the hints for Docker requirements in our .cwl files, this is because we wanted to ensure that each step uses the given .sif file as the default container. CWLTool specifies that this is how the "--default-container" option is to be used, to run any steps without specific containers in that container. Additionally, once we confirm how this works, we would like to use this same setup on a far more complex workflow that currently runs in this way using cwltool and would prefer not to have to modify each individual step in this significantly larger workflow, which is why we were approaching this by altering shell options, rather than individual steps.
If anyone knows how we can force toil-cwl-runner to use the specified singularity image the same way cwltool does, we would really appreciate any information you could share, or help you could provide.
This isn't a regular support channel for Toil or CWL, but luckily I do monitor this tag.
Here's a fix, sorry you ran into this issue! Thanks for the detailed explanation and standalone reproducer.
https://github.com/DataBiosphere/toil/pull/4730
In the future, Toil issues should be submitted to https://github.com/DataBiosphere/toil/issues and general CWL support is available at https://cwl.discourse.group/