datafold

data-diff

Python2022–2025

datafold / data-diff · Archived

data-diff

Compare tables within or across databases

Explore on GitHub Read Hacker News discussions

8Distinct people

10Mentions

0Explicit recommenders

3KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Compare tables within or across databases

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 11:56:56 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year10 dated mentions

Observed mentions: 2022-06-22 15:28:04 UTC to 2025-02-19 00:43:27 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	6	4	2
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	2	0	2
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	1	0	1
2025	2025-01-01 00:00:00 UTC	2026-01-01 00:00:00 UTC	1	0	1

What Hacker News says

10 sources

NortySpock·Feb 19, 2025

As a data engineer, who is regularly fighting - "these two databases have different SQL dialects" - "did we miss a few rows due to poor transaction-isolation when trying to query recently changed rows on the upstream database" - "is there some checksum of a region of cells that accepts any arrangement of rows and columns that doesn't require me to think about ordering?" - "is there a subtle data format conversion problem silently mangling my data?" ...I've toying with trying to find a way to serialize everything consistently into something that can be XOR'd, then compare the output of XOR for two tables in two different databases that should be identical, without having to do some giant order-by comparison. Ideally, if necessary, this could be done by eye with two query windows pointing at two different databases... Basically, Datafold's

Read the rest

datadiff, but in a way that could plausibly be home-rolled for on-premise applications (especially quick spot-checks), and not be a total maintenance nightmare... https://github.com/datafold/data-diff Don't have anything working yet, but it just seems like one could at least xor a bunch of integers and get something useful... Somehow.

In XOR

thenbe·Mar 26, 2024

> we need to easily be able to see diffs between different versions Can git attributes help in this case? It allows you to teach git how to diff binary files using external tools. Here [0] is a demonstration for teaching git to produce an "image diff" for *.png files using exiftool. You can do something similar for *.sqlite files by adding these 3 lines [1] [2]. The sqlite3 cli needs to be installed. Alternatively, there's a tool that might also fit the bill called datafold/data-diff [3]. I'm pretty sure I originally heard of it on a HN thread so those comments may offer even more alternative solutions. [0]: https://youtu.be/Md44rcw13k4?t=540 [the relevant timestamp is @ 9:00] [1]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [2]: https://github.com/kriansa/dotfiles/blob/7a8c1b11b06378b8ca8... [3]: https://github.com/da

Read the rest

tafold/data-diff

In Friends don't let friends export to CSV

poulpi·Jul 28, 2023

If the issue happen a lot, there is also: https://github.com/datafold/data-diff That is a nice tool to do it cross database as well. I think it's based on checksum method.

In How to check if two SQL tables are the same

hichkaker·Feb 1, 2023

I did data engineering for 6 years and am building a company to automate SQL validation for dbt users. First, by “testing SQL pipelines”, I assume you mean testing changes to SQL code as part of the development workflow? (vs. monitoring pipelines in production for failures / anomalies). If so: 1 – assertions. dbt comes with a solid built-in testing framework [1] for expressing assertions such as “this column should have values in the list [A,B,C]” as well checking referential integrity, uniqueness, nulls, etc. There are more advanced packages on top of dbt tests [2]. The problem with assertion testing in general though is that for a moderately complex data pipeline, it’s infeasible to achieve test coverage that would cover most possible failure scenarios. 2 – data diff: for every change to SQL, know exactly how the code change affects the

Read the rest

output data by comparing the data in dev/staging (built off the dev branch code) with the data in production (built off the main branch). We built an open-source tool for that: https://github.com/datafold/data-diff, and we are adding an integration with dbt soon which will make diffing as part of dbt development workflow one command away [2] We make money by selling a Cloud solution for teams that integrates data diff into Github/Gitlab CI and automatically diffs every pull request to tell you the how a change to SQL affects the target table you changed, downstream tables and dependent BI tools (video demo: [3]) I’ve also written about why reliable change management is so important for data engineering and what are key best practices to implement [4] [1] https://docs.getdbt.com/docs/build/tests [2] https://github.com/calogica/dbt-expectations [3] https://github.com/datafold/data-diff/pull/364 [4] https://www.datafold.com/dbt [5] https://www.datafold.com/blog/the-day-you-stopped-breaking-y...

In Ask HN: How do you test SQL?

hichkaker·Dec 15, 2022Submission

Data-diff v0.3: DuckDB, efficient in-database diffing and more

Hi HN: We at Datafold are excited to announce a new release of data-diff (https://github.com/datafold/data-diff), an open-source tool that efficiently compares tables within or across a wide range of SQL databases. This release includes a lot of new features, improvements and bugfixes. We released the first version 6 months ago because we believe that diffing data is as fundamental of a capability as diffing code in data engineering workflows. Over the past few months, we have seen data-diff being adopted for a variety of use-cases, such as validating migration and replication of data between databases (diffing source and target) and tracking the effects of code changes on data (diffing staging/dev and production environments). With this new release data-diff is significantly faster at comparing tables within the same database, especially

Read the rest

when there are a lot of differences between the tables. We've also added the ability to materialize the diff results into a database table, in addition to (or instead of) outputting them to stdout. We've added support for DuckDB, and for diffing schemas. Improved support for alphanumerics, and threading, and generally improved the API, the command-line interface, and stability of the tool. We believe that data-diff is a valuable addition to the open source community, and we are committed to continue growing it and the community around it. We encourage you to try it out and let us know what you think! You can read more about data-diff on our GitHub page at the following link: https://github.com/datafold/data-diff/ To see the list of changes for the 0.3.0 release, go here: https://github.com/datafold/data-diff/releases/tag/v0.3.0

26 points · Open discussion

erezsh·Sep 4, 2022

Looks useful! Do you have a way to validate that the data was copied correctly and entirely? If not, you might want to consider integrating data-diff for that - https://github.com/datafold/data-diff

In Show HN: Open-source infra for building embedded data pipelines

mooreds·Jun 30, 2022Submission

Data-Diff

1 points · Open discussion

feross·Jun 24, 2022Submission

Efficiently diff rows across two different databases

1 points · Open discussion

Sirupsen·Jun 22, 2022

There is plans to support pretty much every database. The reason it’s not supported currently is because its md5 hashing is too slow, so we need to do something different for it, e.g. just sum for types that support it. It’s similar for databases we plan to support that don’t support MD5 too, for example ElasticSearch. If anyone knows a dev on the MSSQL team we could speak to, we’d be eager to be connected See https://github.com/datafold/data-diff/issues/51

In Show HN: Data Diff – compare tables of any size across databases

hichkaker·Jun 22, 2022Submission

Show HN: Data Diff – compare tables of any size across databases

Gleb, Alex, Erez and Simon here – we are building an open-source tool for comparing data within and across databases at any scale. The repo is at https://github.com/datafold/data-diff, and our home page is https://datafold.com/. As a company, Datafold builds tools for data engineers to automate the most tedious and error-prone tasks falling through the cracks of the modern data stack, such as data testing and lineage. We launched two years ago with a tool for regression-testing changes to ETL code https://news.ycombinator.com/item?id=24071955. It compares the produced data before and after the code change and shows the impact on values, aggregate metrics, and downstream data applications. While working with many customers on improving their data engineering experience, we kept hearing that they needed to diff their data across databases t

Read the rest

o validate data replication between systems. There were 3 main use cases for such replication: (1) To perform analytics on transactional data in an OLAP engine (e.g. PostgreSQL > Snowflake) (2) To migrate between transactional stores (e.g. MySQL > PostgreSQL) (3) To leverage data in a specialized engine (e.g. PostgreSQL > ElasticSearch). Despite multiple vendors (e.g., Fivetran, Stitch) and open-source products (Airbyte, Debezium) solving data replication, there was no tooling for validating the correctness of such replication. When we researched how teams were going about this, we found that most have been either: Running manual checks: e.g., starting with COUNT(*) and then digging into the discrepancies, which often took hours to pinpoint the inconsistencies. Using distributed MPP engines such as Spark or Trino to download the complete datasets from both databases and then comparing them in memory – an expensive process requiring complex infrastructure. Our users wanted a tool that could: (1) Compare datasets quickly (seconds/minutes) at a large (millions/billions of rows) scale across different databases (2) Have minimal network IO and database workload overhead. (3) Provide straightforward output: basic stats and what rows are different. (4) Be embedded into a data orchestrator such as Airflow to run right after the replication process. So we built Data Diff as an open-source package available through pip. Data Diff can be run in a CLI or wrapped into any data orchestrator such as Airflow, Dagster, etc. To solve for speed at scale with minimal overhead, Data Diff relies on checksumming the data in both databases and uses binary search to identify diverging records. That way, it can compare arbitrarily large datasets in logarithmic time and IO – only transferring a tiny fraction of the data over the network. For example, it can diff tables with 25M rows in ~10s and 1B+ rows in ~5m across two physically separate PostgreSQL databases while running on a typical laptop. We've launched this tool under the MIT license so that any developer can use it, and to encourage contributions of other database connectors. We didn't want to charge engineers for such a fundamental use case. We make money by charging a license fee for advanced solutions such as column-level data lineage, CI workflow automation, and ML-powered alerts.

127 points · Open discussion