cleanlab

Years ago, we showed the world it was possible to automatically detect label errors in classification datasets via machine learning. Since that moment, folks have asked whether the same is possible for regression datasets? Figuring out this question required extensive research since properly accounting for uncertainty (critical to decide when to trust machine learning predictions over the data itself) poses unique challenges in the regression setting. Today we have published a new paper introducing an effective method for “Detecting Errors in Numerical Data via any Regression Model”. Our method can find likely incorrect values in any numerical column of a dataset by utilizing a regression model trained to predict this column based on the other data features. We’ve added our new algorithm to our open-source cleanlab library for you to algo

Sharing some context here: in grad school, I spent months writing custom data analysis code and training ML models to find errors in large-scale datasets like ImageNet, work that eventually resulted in this paper (https://arxiv.org/abs/2103.14749) and demo (https://labelerrors.com/). Since then, I’ve been interested in building tools to automate this sort of analysis. We’ve finally gotten to the point where a web app can do automatically in a couple of hours what I spent months doing in Jupyter notebooks back in 2019—2020. It was really neat to see the software we built automatically produce the same figures and tables that are in our papers. The blog post shared here is results-focused, talking about some of the data and dataset-level issues that a tool using data-centric AI algorithms can automatically find in ImageNet, which we used as

I built Vizzy as a hackathon project, with the goal of explaining how relatively simple (but clever) algorithms can be a powerful tool to automatically find issues in datasets, including label errors and out-of-distribution data. Vizzy uses a JavaScript port of (a part of) https://github.com/cleanlab/cleanlab, which implements the algorithms described in https://arxiv.org/abs/1911.00068. There are other neat technical nuggets in the implementation of Vizzy as well, including ML model training in the browser (using features from a pretrained ResNet-18, performing truncated SVD, and using an SVM model for speed). If you’re interested in the details of how Vizzy works, check out this blog post: https://cleanlab.ai/blog/cleanlab-vizzy/ I’m happy to answer any questions related to Vizzy, cleanlab, or confident learning and data-centric AI in g

Cleanlab (https://github.com/cleanlab/cleanlab) is a family of algorithms for automatically finding issues in datasets. It might seem surprising that it’s possible to automatically identify label errors and out-of-distribution data; Cleanlab does this using the algorithms published in https://arxiv.org/abs/1911.00068. Cleanlab’s algorithms, while clever, are actually relatively simple. To help myself (and others!) build intuition for how they work, I built Vizzy, an interactive demo that runs in the browser. Vizzy lets you experiment with an example dataset, tweak the labels, and run Cleanlab to automatically find issues like label errors and out-of-distribution data Vizzy includes a JavaScript port of (a part of) cleanlab, along with other neat technical nuggets including ML model training in the browser (using features from a pretrained