Management System (Cromwell vs Nextflow) would be better for the optimization of job execution time?

Question

Which one of the two Management System (Cromwell vs Nextflow) would be better for the optimization of job execution time?

Google Kubernetes(GKE) is used for scheduling and monitoring the jobs.

From what I understood it is possible to integrate both of them, but there is no information about which one is more efficient.

Answer 1

Denisa!

I'm not well versed with Cromwell, so I may miss one or two features that it provides, but on the Nextflow side I can clearly see a few things related to job execution time that are not standard for workflow engines, but Nextflow provides them to you

1. Implicit parallelization: Even if you don't know what parallelization is, and the set of software you're running in your pipeline doesn't take advantage of it, implicitly and automatically Nextflow will try to help you with that. For example, if you have a step in your pipeline that will be applied to many files, Nextflow won't do it sequentially by default. Do X to file 1. Ended? Now do X to file 2. Ended? Now... Nope! It won't do it like that. It will run this step in many files at the same time (you can increase or decrease how many parallel tasks will run concurrently).

2. Nextflow Tower uses Artificial Intelligence to recommend better resource configuration for you, which means not only using cloud resources more efficiently but also spending less money

These are two things related to job execution time with Nextflow that come to mind when I read your question. Maybe there are more

Answer 2

Cromwell developer here. Cromwell does not have a Kubernetes backend. On GCP it uses an abstraction called the Cloud Life Sciences API that spins up its own compute nodes. If running jobs on Kubernetes is a hard requirement, a different product would be best.