cromwell

If you're into workflow runners, Reflow and Cromwell (https://github.com/broadinstitute/cromwell) are the only two really to consider. Having tried them all, these two are by far the best and most supported (and there are 100s!) Cromwell is great because it is google cloud native and supported within the Terra ecosystem (https://app.terra.bio/) meaning you do not need to host it yourself - you can just connect your google account and go. Reflow, I've heard, is a little more "professional" given that the Grail team is heavily ex-Google. But both can scale to massively parallel (1000+ parallel analyses).

We built everything ourselves. The parser in particular was written in-house by a Person Who No Longer Works Here who was a parser enthusiast (Conway’s law). If we were starting from scratch today I would think we would use an off the shelf package. One excellent choice was to make the language [0] an open standard and now there is at least one totally novel implementation [1] in addition to ours [2]. 0. https://github.com/openwdl/wdl 1. https://github.com/chanzuckerberg/miniwdl 2. https://github.com/broadinstitute/cromwell

A very clever way to run genomic analysis on GCP is via WDL ( https://software.broadinstitute.org/wdl/ ) workflows executed on the Cromwell ( https://github.com/broadinstitute/cromwell ) service. Using WDL you define tasks, and compose those into workflows, then Cromwell executes those tasks on GCP using the pipelines api ( https://cloud.google.com/genomics/reference/rest/v1alpha2/pi... ). So you don't need to set up any instances at all, each stage of your pipeline, a task, is executed in its own docker instance, with files from google cloud storage being materialized on that instance for you. When that task is done, outputs are sent to cloud storage buckets, and everything is torn down. Picard can pull reads directly out of the google genomics API, and more of the Broad tools are adopting the GA4GH api, so this means an increasing amoun