kedro

Interesting, thanks for sharing. I'll definitely take a look, although at this point I am so comfortable with Snakemake, it is a bit hard to imagine what would convince me to move to another tool. But I like the idea of composable pipelines: I am building a tool (too early to share) that would allow to lay Snakemake pipelines on top of each other using semi-automatic data annotations similar to how it is done in kedro (https://github.com/kedro-org/kedro).

An often overlooked framework used by NASA among others is Kedro https://github.com/kedro-org/kedro. Kedro is probably the simplest set of abstractions for building pipelines but it doesn't attempt to kill Airflow. It even has an Airflow plugin that allows it to be used as a DSL for building Airflow pipelines or plug into whichever production orchestration system is needed.

Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. It borrows concepts from software engineering best-practice and applies them to machine-learning code; applied concepts include modularity, separation of concerns and versioning. Join users such as NASA, Telkomsel, XP Investimentos, Sber and Beamery! Check out the repository here: https://github.com/kedro-org/kedro/

Kedro, Quantumblack Labs | Python Software Engineer | London | REMOTE currently, ONSITE ? | Full-time Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. It borrows concepts from software engineering best-practice and applies them to machine-learning code; applied concepts include modularity, separation of concerns and versioning. Kedro is listed on the 2020 ThoughtWorks Technology Radar (https://mattturck.com/data2020/) and has been framed as the "React for Data Science". Check out our repository: https://github.com/quantumblacklabs/kedro Please apply here mentioning Kedro + HN in your application: https://rb.gy/1njxa1 We are at an exciting juncture in the maturing ML and MLOps space. We are keen to expand our function set through more sophisticated deployments and integrations

Kedro, Quantumblack Labs | Python Software Engineer | London | REMOTE currently, ONSITE ? | Full-time Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code. It borrows concepts from software engineering best-practice and applies them to machine-learning code; applied concepts include modularity, separation of concerns and versioning. Kedro is listed on the 2020 ThoughtWorks Technology Radar (https://www.thoughtworks.com/radar/languages-and-frameworks/...) and the 2020 Data & AI Landscape (https://mattturck.com/data2020/) and has been framed as the "React for Data Science". Check out our repository: https://github.com/quantumblacklabs/kedro Please apply here mentioning Kedro + HN in your application: https://rb.gy/1njxa1 We are at an exciting juncture in the maturing ML and MLOps spa

If you are feeling overwhelmed with yet another machine learning pipeline automation framework, you should check out Kedro (https://github.com/quantumblacklabs/kedro). Kedro has the simplest, leanest, functional-programming inspired pipeline definition and also spits out AirFlow and other formats readily + comes in with an integrated visualisation framework which is stunning & effective.

One way to get around this is to use Kedro https://github.com/quantumblacklabs/kedro, which is the most minimal possible pipeline interface, yet allows you to export to other pipeline formats and/or build your own exporters.

Airflow is an incredibly powerful framework to use in production, but a little unweildy for anything else. You can use something like Kedro (https://github.com/quantumblacklabs/kedro) to get started building pipelines with pure Python functions. Kedro has its own pipeline visualiser and also has an Airflow plugin that can automatically help you generate airflow pipelines from Kedro pipelines. https://github.com/quantumblacklabs/kedro-airflow

> Is it a processing solution? Not really, since it suggests you can offload the heavy lifting to an engine, eg. spark. An orchestrator? Apparently not, because that's a complementary product. So... it's like, a configuration management tool? I actually had the same questions when I was first introduced to Kedro! In my case, I didn't understand the value proposition over something like Apache Beam. After using it, I feel like Kedro provides: 1. a consistent structure across analytics pipelines. It's easy to start and pick up other Kedro projects after you've used it once. 2. convenient and consistent I/O via the data catalog. The fact that we can configure and swap out data sources at ease is a huge plus, and we also rely heavily on data versioning. 3. easy integration with existing frameworks (PySpark, vanilla P