redun

Completely agree with this view. I wish I had seen this thesis earlier when developing redun (https://github.com/insitro/redun/). Looks like a lot of very useful ideas for defining such a system. We based redun's execution model on very similar ideas of functional programming and graph reduction. In addition, we made it work as an embedded DSL within Python, so one can easily use all the typical data science and ML libraries in a workflow. This has been very helpful for building biotech workflows (genomics, imaging, chem). I am a bit surprised why many workflow systems shy away from full turing-complete. You usually don't need to trade that away for automatic parallelism, caching, etc.

Interesting! I feel like this is a cross between https://github.com/dagworks-inc/burr (switch state for context) and https://github.com/Netflix/metaflow because the output of the "task" declares its next hop... The challenge with this approach though is that you need to run the actual code to see what it does, or as a developer build up a mental model of the code ... but it does shine in certain use cases -- and also reminds me of https://github.com/insitro/redun because it takes this approach too.

I'm personally a huge fan of redun¹ for running computational pipelines. It's pure python, it's easy to learn/debug, it has automatic caching, retry, provenance logging, and a great integration with AWS Batch for running large jobs. I've been really impressed with how easy it is to run a job to completion that fans out to thousands of AWS spot instances at once. I've used nextflow in the past, and I've found it to be much harder to use. Learning another DSL is annoying, documentation was sparse, I constantly ran into bugs, and it was hard to debug in general. I don't know how much it's changed over the past 3 years though. ¹https://github.com/insitro/redun