splink

Splink is a notable endeavor in this regard from the MoJ. https://github.com/moj-analytical-services/splink.

Just recently released version 4 of our free library for data deduplicating/linkage at scale. After four years working on it, finally getting to the stage where I'm really happy with both the performance and the API design. https://github.com/moj-analytical-services/splink

For deduplicating people, the Fellegi Sunter model is also a powerful approach. Splink[0] is a free Python library that implements this for big datasets. Probably you could combine parts of both approaches as well. Full disclosure: I am the lead author. I've also written up some interactive tutorials on how the method works [1] if anyone's interested [0]https://github.com/moj-analytical-services/splink [1]https://www.robinlinacre.com/intro_to_probabilistic_linkage/

Thanks, appreciate it. Fastlink was the inspiration for Splink - the fundamental statistical model is very similar. The first version of Splink was essentially a port to make it work faster and at greater scale, but we've subsequently added quite a bit of additional functionality Feel free to ask questions if you run into any issues - we're usually fairly good at responding: https://github.com/moj-analytical-services/splink/discussion...

Naive Bayes is also surprisingly powerful for deduplication of large datasets of messy data (and the related problem of record linkage). It's still competitive with cutting edge approaches if carefully applied, and usually much faster so it can be applied to big data. It's the engine behind Splink, the free Python library that I develop: https://github.com/moj-analytical-services/splink

Splink, a Python library for entity resolution (also known as record linkage, data matching). It's free, it works at scale, and is compatible with multiple database backends, and has a suite of interactive visualisations: https://github.com/moj-analytical-services/splink I believe it is the fastest and most accurate free solution

If you're curious, I've written a FOSS record linkage library that executes everything as SQL. It supports multiple SQL backends including DuckDB and Spark for scale, and runs faster than most competitors because it's able to leverage the speed of these backends: https://github.com/moj-analytical-services/splink

Splink - a FOSS python library for probabilistic record linkage (fuzzy matching/entity resolution). Splink is dramatically faster and works on much larger datasets than other open source libraries. I'm particularly proud of the fact we support multiple execution backends (at the moment, DuckDb Spark Athena and Sqlite, but additional adaptors are relatively straightforward to write). We've had >4 million pypi downloads and it's used in government, academia and the private sector, often replacing extremely expensive proprietary solutions. https://github.com/moj-analytical-services/splink More info in blog posts here: https://www.robinlinacre.com/introducing_splink/ https://www.robinlinacre.com/splink_3/

SQLGlot is great. We've used it to extend our FOSS probabilistic data linking library[1] so that it is now capable of executing against a variety of SQL backends (Spark, Presto, DuckDB, Sqlite), significantly widening our potential user base. We implement the core statistical model in SQL, and then use SQLGlot to transpile to the target execution engine. One big motivation was to futureproof our work - we're no longer tied down to Spark, and so when the 'next big thing' (GPU accelerated SQL for analytics?) comes along, it should be relatively straightforward to support it by writing another adaptor. Working on this has highlighted some of the really tricky problems associated with translating between SQL engines, and we haven't hit any major problems, so kudos to the author! [1] https://github.com/moj-analytical-services/splink/tree/spl

This is great. In terms of real-world uses, I'm currently working on enabling DuckDB as a backend in Splink[1], software for record linkage at scale. Central to the software is an iterative algorithm (Expectation Maximisation) that performs a large number of group-by aggregations on large tables. Until recently, it was PySpark only, but we've found DuckDB gives us great performance on medium size data. This will be enabled in a forthcoming release (we have an early pre-release demo of duckdb backend[2]). This new DuckDB backend will probably be fast enough for the majority of our users, who don't have massive datasets. With this in mind, excited to hear that: > Another large area of future work is to make our aggregate hash table work with out-of-core operations, where an individual hash table no longer fits in memory, this is particula