HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
pditommaso
awesome-pipeline
Learning & Reference2015–2023
pditommaso / awesome-pipeline

awesome-pipeline

A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin

7Distinct people
9Mentions
1Explicit recommenders
6.6KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

As a long-time fan of DAG-oriented tools, congrats on the launch. Maybe you can get added here https://github.com/pditommaso/awesome-pipeline now or in the future... This is a…

slotrans used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 12:01:57 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year9 dated mentions

Observed mentions: 2015-08-03 20:09:05 UTC to 2023-03-07 23:16:04 UTC.

0122015-01-01 00:00:00 UTC to 2016-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20152016-01-01 00:00:00 UTC to 2017-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2017-01-01 00:00:00 UTC to 2018-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments2019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 2 mentions; 0 submissions, 2 comments20192020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments2023

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20152015-01-01 00:00:00 UTC2016-01-01 00:00:00 UTC110
20162016-01-01 00:00:00 UTC2017-01-01 00:00:00 UTC000
20172017-01-01 00:00:00 UTC2018-01-01 00:00:00 UTC202
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC202
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC202
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC101
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC000
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC000
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC101

What Hacker News says

2 sources
+1 to this comment. It is a reasonably sized body of research, too. On top of this, scientific workflow scheduling maps to the classic scheduling problem, which is known to be NP-hard - the fact that the authors infer that automating the traversal is all that is necessary suggests that they have an incomplete understanding of the problem at hand. For those interested, a nice list of workflow managers, engines, and even languages can be found at https://github.com/pditommaso/awesome-pipeline.
In Modeling Science as a Directed Graph
SciLuigi adds a simple level of audit logging to Luigi [1] (See the AuditTrailHelpers mixin in https://github.com/pharmbio/sciluigi/blob/master/sciluigi/au... for details), together with an API that makes it much easier to work do changes to the workflow dependency graph, when you're in development/exploratory phase. Personally moved to improve on the ideas in SciLuigi, but replacing Luigi with Go's concurrency primitives, in SciPipe [2]. In SciPipe, an accompanying ".audit.json" file, containing the full history of commands and parameters used to create every single output file of the workflow (even intermediate ones), to help keep track of how each file was produced. Among lightweight solutions, the popular ones these days seem to be NextFlow [3], Snakemake [4], BPipe [5] and others. You'd really have to check out the "awesome pipelines
Read the rest
" [6] list, to get any kind of overview. When more infrastructure setup is feasible, I think PachyDerm [7] (mentioned elsewhere in the thread) looks really powerful with its "Git for data" approach. Something I'd wish to use as an overarching solution within which to run my SciPipe workflows. [1] https://github.com/pharmbio/sciluigi [2] http://scipipe.org [3] http://nextflow.io [4] https://bitbucket.org/snakemake/snakemake/wiki/Home [5] http://bpipe.org [6] https://github.com/pditommaso/awesome-pipeline [7] http://pachyderm.io
In Data Science Workflow: Overview and Challenges

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.