hopsworks

We built the first open-source feature store for ML, https://github.com/logicalclocks/hopsworks , when every existing proprietary feature store (Uber Michelangelo and Bighead at AirBnb) were shouting about how their DSL for feature engineering was the future. Fast-forward 2 years and it is clear that Data Scientists want to work with Python, not with a DSL. We based our Feature Store on a Dataframe API for Python/PySpark. The DSL can never evolve at the same rate as libraries in a general-purpose programming language. So, your DSL is great for show-casing a Feature Store, but when you need to compute embeddings or train a GAN or done any type of feature engineering that is not a simple time-window aggregation, you pull out Python (or Scala/Java). I am old enough to have seen many DSLs in different domains (GUIs, aspect-oriented programmin

You should checkout Hopsworks (disclaimer: work on it). It does all of the above (including a Feature Store, notebooks as jobs, airflow for ML pipelines, model serving (TensorFlowServing, Flask, and soon KFserving), experiments, a project-based multi-tenancy model that supports sensitive data on a shared cluster, and a UI. It does not have a labelling system - but you can pip install libraries. You don't need to learn Docker - each project has a conda environment, and we compile Docker images for projects transparently, so jobs are tied to Docker images, but you don't need to write a Dockerfile (this is a huge win for data scientists). You can run Python jobs (connect to a k8s cluster) or Spark/Flink jobs (on Hopsworks itself). Open-source: * https://github.com/logicalclocks/hopsworks Managed platform on AWS/Azure (with elastic compute/s

But is it open-source? Our Hopsworks Feature Store is * https://github.com/logicalclocks/hopsworks as is GoJEK Feast * https://github.com/feast-dev/feast I really believe that all Enterprise platform software should have an open-source version if it is to have a meaningful effect on how people work (in this case, how Data Scientists and Data Engineers work together).