metaflow

Stay tuned! We have some cool new features coming soon to support agentic workloads (teaser: https://github.com/Netflix/metaflow/pull/2473) If you are curious, join the Metaflow Slack at http://slack.outerbounds.co and start a thread on #ask-metaflow

Interesting! I feel like this is a cross between https://github.com/dagworks-inc/burr (switch state for context) and https://github.com/Netflix/metaflow because the output of the "task" declares its next hop... The challenge with this approach though is that you need to run the actual code to see what it does, or as a developer build up a mental model of the code ... but it does shine in certain use cases -- and also reminds me of https://github.com/insitro/redun because it takes this approach too.

as well as to ML - Netflix Prize (https://en.wikipedia.org/wiki/Netflix_Prize) and Metaflow (https://github.com/Netflix/metaflow)

This is very cool! If you are interested in using a similar setup for your ML workloads (Argo + K8S), I'd love to chat with you (ville@outerbounds.co). We are working on integrating Argo & K8S for Metaflow, an open-source ML framework originally developed at Netflix https://github.com/Netflix/metaflow/pull/434

It is a bit of a hidden gem but Metaflow includes a Boto-based highly parallelized, error-tolerant S3 client that Netflix uses routinely to get 10-20Gbps throughput between EC2 and S3. Technically it is independent from Metaflow, so you could use it as a stand-alone, high-performance S3 client. See docs here https://docs.metaflow.org/metaflow/data#store-and-load-objec... And code here https://github.com/Netflix/metaflow/tree/master/metaflow/dat... (I wrote it originally - AMA if curious)

has anyone done a comparison of ML pipelines from a devops centric perspective ? For example, Metaflow doesnt support kubernetes today - https://github.com/Netflix/metaflow/issues/16 so ultimately the scale up story in most of these management tools is iffy. I previously asked about kubeflow here - https://news.ycombinator.com/item?id=24808090 . Seems people think its pretty "horrendous". It seems most of these tools assume a very specialised devops team who will work around the ml tool...rather than the ml tool making this easy.

Congrats on the launch! This looks interesting, however I feel like this space is quite crowded. You mentioned that your most important feature is the fact that you are open-source, but off the top of my head I can think of several projects: * Kubeflow: https://github.com/kubeflow/kubeflow * MLFlow: https://github.com/mlflow/mlflow * Pachyderm: https://github.com/pachyderm/pachyderm * DVC: https://github.com/iterative/dvc * Polyaxon: https://github.com/polyaxon/polyaxon * Sacred: https://github.com/IDSIA/sacred * pytorch-lightning + grid: https://github.com/PyTorchLightning/pytorch-lightning * DeterminedAI: https://github.com/determined-ai/determined * Metaflow: https://github.com/Netflix/metaflow * Aim: https://github.com/aimhubio/aim * And so many more... In addition to this list, several other hosted platform offer experiments trackin

Good question. We have many common features covered: - Container management: See https://docs.metaflow.org/metaflow/dependencies - Search for artifacts: see https://docs.metaflow.org/metaflow/client - Automatic publishing of web apps: we have this internally but it is not open-source yet. If it interests you, react to this issue (https://github.com/Netflix/metaflow/issues/3) Let us know if you notice any other interesting features missing! Feel free to open a GitHub issue or reach out to us on http://chat.metaflow.org

Good question! A typical Metaflow workflow at Netflix starts by reading data from our data warehouse, either by executing a (Spark)SQL query or by fetching Parquet files directly from S3 using the built-in S3 client. We have some additional Python tooling to make this easy (see https://github.com/Netflix/metaflow/issues/4) After the data is loaded, there are bunch of steps related to data transformations. Training happens with an off-the-shelf ML library like Scikit Learn or Tensorflow for training. Many workflows train a suite of models using the foreach construct. The results can be pushed to various other systems. Typically they are either pushed to another table or as a microservice (see https://github.com/Netflix/metaflow/issues/3)

Hey I meant to track one of y'all down at the MLOps conference, but didn't get the chance. I've built a very shitty version of a cached-execution DAG thing internally, and one of the design decisions I made was to have it so that parent nodes in the DAG don't need to know anything about child nodes. This allows for larger DAG builders to be more easily subclassed. MetaFlow doesn't do that -- instead each 'step' has to know what to call next, which means that if I wanted to subclass e.g. the MovieStatsFlow in [here](https://github.com/Netflix/metaflow/blob/master/metaflow/tut...) and say, add some sort of input pre-processing before the compute_statistics call, I'd essentially end up having to either override what compute_statistics does to not match its name _or_ copy-past e that first step just to replace that last line. I'm sure this de

Yes indeed. You can access your model via the Metaflow client inside your service - https://docs.metaflow.org/metaflow/client. We are looking into releasing the hosting component on Metaflow in the future - https://github.com/Netflix/metaflow/issues/3

How is this different / better to existing tools or workflows? I don't like to criticise new frameworks / tools without first understanding them, but I like to know what some key differences are without the marketing/PR fluff before giving one a go. For instance, this tutorial example here (https://github.com/Netflix/metaflow/blob/master/metaflow/tut...) does not look substantially different to what I could achieve just as easily in R, or other Python data wrangling frameworks. Is the main feature the fact I can quickly put my workflows into the cloud?

Hi bhtucker, We do have plans to integrate with production-grade schedulers in the very near future. https://github.com/Netflix/metaflow/issues/2