argo-workflows

Like Argo Workflows? https://github.com/argoproj/argo-workflows

There are too many different tools in the space. I've been heavily researching workflow / ETL frameworks this week, and even after culling the ones that seemed like poor fits, I'm still left with: - https://github.com/getpopper/popper - https://docs.pachyderm.com/ - https://github.com/lyft/flyte - https://aws.amazon.com/step-functions/ - https://github.com/spotify/luigi - https://docs.metaflow.org/ - https://github.com/dagster-io/dagster - https://github.com/argoproj/argo - https://github.com/prefecthq/prefect

Have a look at Argo if you are interested in leveraging k8s infrastructure https://github.com/argoproj/argo/blob/master/README.md#what-.... For us, we settled with Netflix Conductor as it scaled pretty well and allowed us to have pretty complex workflows and error paths and retry logic. Also is an independent and standalone tech.

This looks great. To echo other folks’ sentiment - I think you mean CI-like pipelines specifically, though it could be extended to do some other stuff. You probably want to list some concrete use cases on the main site (e.g. CI, cron jobs, ETL?). I’m curious what differences/trade offs there are in Gaia vs something like Argo (https://github.com/argoproj/argo) or Buildkite (https://buildkite.com/). It looks like at least one difference is an actual API for steps rather than bash commands. Is there anything interesting in terms of cross job caching (e.g. saving npm install data) or persistent runners? And obviously - what else am I not thinking about?

Another cool Argo: https://github.com/argoproj/argo "The Workflow Engine for Kubernetes"

Well you can use https://github.com/argoproj/argo for workflow needs without need to version everything. For data versioning if you need, you can use https://dvc.org/ As this gives me freedom to choose as if I want data versioning and/or workflow rather than one bug bundle.

Instead of using docker host you might run docker daemon in sidecar container. Here is a sample argo workflow which uses this technique: https://github.com/argoproj/argo/blob/master/examples/sideca...

Docs have been updated and linked to in the getting started guide: https://github.com/argoproj/argo/blob/master/docs/workflow-r...

I work on Argo. Those instructions are mainly for demo purposes because many of the examples need admin privileges. The amount of access which is needed, is really dependent on what your workflow is doing (e.g. if your workflow needs to deploy a resource, then you need to give the service account 'create' privileges on that resource). Currently, the bare minimum for a workflow to function is outlined here: https://github.com/argoproj/argo/issues/983#issuecomment-435... I will make sure that the docs are updated, so that this information is well understood.

And right below that it says: >NOTE: You can also submit workflows which run with a different service account using: >argo submit --serviceaccount <name> To which there is further documentation as to the exact permissions you need. While I only spent about 30 seconds searching, this gets you 90% of the way there from what I can tell on my phone. https://github.com/argoproj/argo/issues/725

Yes, though the problems are real. In its current form Kubernetes can only handle a cluster size of 5,000 nodes: https://kubernetes.io/docs/setup/cluster-large/ and high-churn batch workloads don't work well. Kubernetes does support elastic resource sharing (assuming I understand what they need) via horizontal pod scaling and the cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster.... Although there are rough edges here, I was surprised by the quality of the code in the cluster autoscaler. I kept thinking of new edge cases and expected it to fall over, only to find out that they'd thought of that problem and it handled it gracefully. For batch workloads something like Argo can help: https://github.com/argoproj/argo. So basically I agree, it probably would've made more sense to build something on top of Kuberne

Happy to answer any questions about how we use Argo (https://github.com/argoproj/argo), PlanetUtils (https://github.com/interline-io/planetutils), or any of the other open-source "ingredients."

The latest Argo release (https://github.com/argoproj/argo/releases) supports resubmitting workflows with "memoized" steps.

We developed and use Argo (https://github.com/argoproj/argo), a Kubernetes-native workflow engine. Argo is currently used by companies like Cyrus Biotechnology, Gladly, Google, Intuit, and NVIDIA. Currently collecting use cases and requirements on a Kubernetes-native eventing framework for Argo (https://github.com/argoproj/argo-events/issues/1) to make it easier to kick off workflows.

The Argo project is a workflow engine built on top of kubernetes. Workflows are written as yaml templates and support DAGs as well as loops and conditionals. https://github.com/argoproj/argo We use this at my company to stitch together various scientific software packages each of which can take minutes to 10s of hours to run. Argo supports retrying, resubmitting, suspending, and resuming workflows. It really is a neat project, especially if you are already using kubernetes!