SCscottious·May 5, 2016
I created the WDL specification and parser. You're right that a "workflow" is a squishy term that could mean different things to different people and the lowest common denominator definition is very broad. WDL attempts to give at least one formal definition of what a workflow is. Maybe using terms like "WDL workflow" could make it more concrete.
And to answer your question about if these languages are actually used... absolutely! We use WDL extensively, but it is relatively new, only about 1 year old. We've had a lot of success using it so far.
What does WDL bring to the table? Well, WDL is essentially Makefile on steroids, designed for cloud computing. We want to be able to take a shell script and turn it into a function with parameters and return values. Like this: https://github.com/broadinstitute/wdl/blob/gvda_add_scripts_....
Read the rest
WDL calls this a "task".
Then, once we define a bunch of tasks, we need to be able to link them together. Like, have a sub-set of one WDL task's outputs be the inputs to another task. WDL calls this a "workflow". Here is an example of invoking the above task: https://github.com/broadinstitute/wdl/blob/gvda_add_scripts_...
The workflows we're creating are far too large to be run on a single machine, so we wanted a way to specify what each task needs in terms of CPU/memory/disk so we can provision machines that are capable of running each task. Parallelism is also a very important aspect of WDL, and all workflow languages/engines.
As others have said, this is not a new idea. WDL's primary goal is to make this as easy as possible for people to read, write, maintain, and extend workflows. We knew that these "workflows" would be code and they'd evolve, they'd have to be code reviewed and diff'd, and there'd be pull requests of WDL files, so we needed a way of describing them that would work well in this environment.