plumbing

This reminds me a bit of a Clojure library called Plumbing (formerly Graph): https://github.com/plumatic/plumbing. It also let you create a DAG for structured computation. It was used for a web service, at that time.

Kedro puts emphasis on seamless transition to prod without jeopardizing work in experimentation stage: - pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing - sequential and parallel runners are built-in (don't have to rely on Airflow) - io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn - flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl - there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements

we are using it at event fabric and we love it: http://eventfabric.com/ some libs we use: * immutant http://immutant.org/ * prismatic's plumbing https://github.com/Prismatic/plumbing * korma http://www.sqlkorma.com/ * friend https://github.com/cemerick/friend * metrics clojure: https://github.com/sjl/metrics-clojure/

We're working on it! We removed the software because it needed extensive cross-project reorganization, consolidation, and cleanup which just wasn't possible to do with a clean upgrade path using available resources (3 backend engineers). Plans are to re-release all of this and more, and we've already started with Plumbing and Graph: https://github.com/prismatic/plumbing. Which librar(ies) in particular are you most interested in?

Also, here's a direct link to the source: https://github.com/prismatic/plumbing and a literate test with lots of real examples: https://github.com/Prismatic/plumbing/blob/master/test/plumb... We're also working on other kinds of compilation, including 'direct' ones that compile directly to a single fn, and 'async' ones that handle asynchronous functions and are smarter about spinning up threads: https://gist.github.com/w01fe/4710008