drake

It looks like maak does not have a mechanism for specifying file dependencies (run recipe when dependent file is newer than output file). I'd love to be wrong. I try to use Makefile to document data pipelines, but it's easy for it to become spaghetti. [Edit: There's the decade-stale https://github.com/factual/drake "Drake is similar to GNU Make, but designed especially for data workflow management". I can't remember why I bounced off it.] For command line runners like maak and just, why not use small cli scripts (`quick-math 23 42` instead of `maak quick-math 23 42`)? I think I'm missing the bigger picture.

Drake is nice for these kinds of data dependencies. https://github.com/Factual/drake

Make is often brought out for data, "single machine ETL" jobs, but for big, complicated (and iterative) workflows it doesn't feel good enough to me. What do you folks use? Drake, "make for data" https://github.com/Factual/drake seems ok, but doesn't have "batch" jobs, (aka "pattern rules") where you can do every file in a directory matching a pattern. Others have come up with different swiss army knives but nothing ever sticks for me, it usually ends up as a single Makefile with eg 3 targets that call a bunch of shell scripts. The whole thing would be configurable to build from scratch, but not well set up to do incremental ETL on a per file basis, after I eg delete some extraneous rows in one file, clean up a column, redownload a folder, or add files to a dataset.

To amend, why I'm even bringing this up, what worries me about Drake is this: https://github.com/Factual/drake/pulse/monthly Its GitHub pulse is - dead; For two years now. Makes me think one of the other projects listed might be better choices.

Given the number of tools around, and as this book promotes Drake, has anyone got "comparative experience" with some of the following tools: Cookiecutter: https://drivendata.github.io/cookiecutter-data-science DataVersionControl: https://dataversioncontrol.com/ Drake: https://github.com/Factual/drake Luigi: https://github.com/spotify/luigi Pachyderm: http://www.pachyderm.io/ Sacred: https://github.com/IDSIA/sacred They all focus in slightly different ways on the issue of managing data science/machine learning workflows, so I wonder if someone has a clear preference for one of those over any another. EDIT: added Luigi

I have briefly used https://github.com/Factual/drake for this. It is a bit unpolished and still a work in progress, but has some features that are useful for data science workflows that most software build tools do not cover: * Steps can produce multiple named outputs (for example full.csv and summary.csv), dependency graph can branch in both ways, * Outputs that are directories with many files, * Built-in input/output file sharding (for example by date or month) and HDFS support, * Lots of control over which steps to execute: not just "rebuild target X and any required upstream dependencies". Can select a target and all that depend on it downstream (if you know that a remote file changed), skip rebuilding some steps, etc, * Can put very short (text processing) scripts directly in the Drakefile in Python, Ruby.

Amen right back at ya! (I love the O'Reilly book cover.) I highly recommend people read your blog post. And there's also this classic: https://aadrake.com/command-line-tools-can-be-235x-faster-th... I'll also take this opportunity to plug Make and Drake for manipulating data in a replicable way: https://bost.ocks.org/mike/make/ https://github.com/Factual/drake If you're processing data using tools that cannot trace their ancestry directly to some time before 1985, you're probably wasting your own and your colleagues' time.

How is this different from drake? drake - Data workflow tool, like a "Make for data" (in Clojure): https://github.com/Factual/drake

https://github.com/Factual/drake from the data science toolbox is useful. "Drake is similar to GNU Make, but designed especially for data workflow management"

drake[0] from factual falls into this category as well. [0] - https://github.com/Factual/drake

There are command line tools available to help the transition from 'hack' one liner to a more maintainable / supportable solution. For instance drake (https://github.com/Factual/drake) a 'Make for data' which does dependency checking would allow for sensible restarts of the pipeline. The O'Reilly Data Science at the Command Line book (linked elsewhere in the comments) has a good deal to say on the subject: turning one liners into extensible shell scripts, using drake, using Gnu Parallel.

A lot of the nitty-gritty data munging and processing often gets discarded after a project or never included in the project repo in a meaningful way. I like Drake [0] because we used it a lot at Factual and it really made data generation and formatting very repeatable and easy. I really think the packaging system the author is going for would be best built on top of Drake or a similar workflow management program. Instead of following their laundry list of configuration steps one could manage that automatically with a source-controlled workflow. Drake does have the advantages of non-linear and async workflows being pretty easy to build, maintain, and update. What I would love to see is a data package manager that downloads the raw data and processing workflow, updates any software packages needed to run the workflow, and then spits out

nice post! i'd like to make a shameless plug for Drake, a 'make for data' workflow tool. https://github.com/Factual/drake

Drip now works with Drake! Yes, it's still less than ideal if you're calling Drake hundreds of times from an automated script which you need to run quickly, but for interactive development, it should work just fine: https://github.com/Factual/drake/wiki/Faster-startup:-Drake-...