reflow

Reflow is implemented in Go, but it can be used to run programs in any language. The Reflow language (.rf files) is a DSL, the language is described in more detail here: https://github.com/grailbio/reflow/blob/master/LANGUAGE.md

Reflow [1] is a similar attempt at a slightly different domain: bioinformatics and ETL pipelines. Reflow exposes a data model and programming model that reclaims programmability in these systems, and, by leaning on these abstractions, gives the runtime much more leeway to do interesting things. It unties the hands of the implementer. [1] https://github.com/grailbio/reflow

It seems Reflow falls in this category: https://github.com/grailbio/reflow > Reflow thus allows scientists and engineers to write straightforward programs and then have them transparently executed in a cloud environment. Programs are automatically parallelized and distributed across multiple machines, and redundant computations (even across runs and users) are eliminated by its memoization cache. Reflow evaluates its programs incrementally: whenever the input data or program changes, only those outputs that depend on the changed data or code are recomputed.

Fun fact: Grail (or, Marius at Grail) wrote their own scientific workflow engine in Go, with a focus on AWS as execution platform: https://github.com/grailbio/reflow Fo anyone interested, we reviewed it including a comparison to other Go based wf tools (the number is obviously growing): http://gopherdata.io/post/more_go_based_workflow_tools_in_bi...

> I want an Airflow that runs and scales in the cloud I'd encourage you to look at Reflow [1] which takes a different approach: it's entirely self-managing: you run Reflow like you would a normal programming language interpreter ("reflow run myjob.rf") and Reflow creates ephemeral nodes that scale elastically and that tear themselves down, only for the purpose of running the program. > has extensive observability (monitoring, tracing) Reflow includes a good amount of observability tools out of the box; we're also working on integrating tracing facilities (e.g., reporting progress to Amazon x-ray). > has a full API, and maybe some clear way to test workflows. Reflow's approach to testing is exactly like any other programming language: you write modules that can either be used in a "main" program, or else be used in tests. [1] https://githu

Reflow [1] is also well-suited for ETL workloads. It takes a different tack: it presents a DSL with data-flow semantics and first-class integration with Docker. The result is that you don't write graphs, instead you just write programs that, due to their semantics, can be automatically parallelized and distributed widely, all intermediate evaluations are memoized, and programs are evaluated in a fully incremental fashion: [1] https://github.com/grailbio/reflow