CD4-csaw

A while back I went through several of these kinds of build systems. The one I eventually settled on was Snakemake[1], with which I finally was able to put together an entire complex bioinformatics workflow[2]. I would say that Snakemake's strength is one of Perl's old mottos: "making easy things easy and hard things possible." On the easy side, writing a simple rule with fixed inputs and outputs is just as easy as Make (trading the weird whitespace rules of Make for the less weird whitespace rules of Python). On the hard side, it supports stuff like computing inputs at runtime, or using regex capture groups to determine output file names from input file names. So for example you can have your rule for LaTeX to PDF conversion read the source TeX file, determine all the external images, bib files, etc. referenced from it, and add those fil

I'm using RStudio notebooks heavily in my latest bioinformatics analysis pipeline. They're a great way to produce an HTML report containing code, exposition, results, and plots all in one place. https://github.com/DarwinAwardWinner/CD4-csaw (look at scripts/*.Rmd)

I use snakemake quite a bit, it was cool to scan through your Snakefile and learn some things. The processify decorator looks really useful[0]. It's possible that you could use snakemake subworkflows [1] for this issue of "pre-workflow" workflows. [0] https://github.com/DarwinAwardWinner/CD4-csaw/blob/master/pr... [1] https://bitbucket.org/snakemake/snakemake/wiki/Documentation...

Here's one example where I had to use a kind of ugly hack ot make it work with Snakemake, a Python Makefile-style "DAG-of-rules" workflow tool: https://github.com/DarwinAwardWinner/CD4-csaw Basically, I need to first fetch the metadata on all the samples, and then later group them by treatment based on that metadata. In other words, the structure of later parts of the DAG depends on the results of executing earlier parts of the DAG, so the full structure of the DAG is not known initially. The solution I used was to split the workflow in two: a "pre-workflow workflow" that fetches the sample metadata and then the main workflow which reads the metadata and builds the DAG based on it. See here: https://github.com/DarwinAwardWinner/CD4-csaw/blob/master/Sn... This a common pattern that I see when putting together bioinformatics workflows: the