SHshcheklein·Nov 4, 2024Submission
DataChain: DBT for Unstructured Data
163 points · Open discussiondatachain-ai
datachain
Python2024–2024
datachain-ai / datachain
datachain
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
5Distinct people
17Mentions
0Explicit recommenders
2.8KGitHub stars
What to know before exploring
What it is
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 1:49:46 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by week17 dated mentions
Observed mentions: 2024-07-23 18:12:38 UTC to 2024-11-04 17:34:57 UTC. Weeks start Monday, UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Jul 22 | 2024-07-22 00:00:00 UTC | 2024-07-29 00:00:00 UTC | 2 | 2 | 0 |
| Jul 29 | 2024-07-29 00:00:00 UTC | 2024-08-05 00:00:00 UTC | 0 | 0 | 0 |
| Aug 5 | 2024-08-05 00:00:00 UTC | 2024-08-12 00:00:00 UTC | 3 | 2 | 1 |
| Aug 12 | 2024-08-12 00:00:00 UTC | 2024-08-19 00:00:00 UTC | 2 | 1 | 1 |
| Aug 19 | 2024-08-19 00:00:00 UTC | 2024-08-26 00:00:00 UTC | 2 | 1 | 1 |
| Aug 26 | 2024-08-26 00:00:00 UTC | 2024-09-02 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2 | 2024-09-02 00:00:00 UTC | 2024-09-09 00:00:00 UTC | 1 | 1 | 0 |
| Sep 9 | 2024-09-09 00:00:00 UTC | 2024-09-16 00:00:00 UTC | 0 | 0 | 0 |
| Sep 16 | 2024-09-16 00:00:00 UTC | 2024-09-23 00:00:00 UTC | 1 | 1 | 0 |
| Sep 23 | 2024-09-23 00:00:00 UTC | 2024-09-30 00:00:00 UTC | 1 | 1 | 0 |
| Sep 30 | 2024-09-30 00:00:00 UTC | 2024-10-07 00:00:00 UTC | 1 | 1 | 0 |
| Oct 7 | 2024-10-07 00:00:00 UTC | 2024-10-14 00:00:00 UTC | 0 | 0 | 0 |
| Oct 14 | 2024-10-14 00:00:00 UTC | 2024-10-21 00:00:00 UTC | 3 | 1 | 2 |
| Oct 21 | 2024-10-21 00:00:00 UTC | 2024-10-28 00:00:00 UTC | 0 | 0 | 0 |
| Oct 28 | 2024-10-28 00:00:00 UTC | 2024-11-04 00:00:00 UTC | 0 | 0 | 0 |
| Nov 4 | 2024-11-04 00:00:00 UTC | 2024-11-11 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
17 sourcesRight, DVC caches data for consistency and reproducibility.
If caching is not needed and streaming required, we've created a sister tool DataChain. It's even supports WebDataset and can stream from tar archives and filter images by metadata.
WebDataset example: https://github.com/iterative/datachain/blob/main/examples/mu...
In Data Version Controlhi there! Maintainer and author here. Excited to see DVC on the front page!
Happy to answer any questions about DVC and our sister project DataChain https://github.com/iterative/datachain that does data versioning with a bit different assumptions: no file copy and built-in data transformations.
In Data Version ControlDataChain: Pythonic AI Data Warehouse
1 points · Open discussionTool to prepare, curate, version datasets for AI/ML
1 points · Open discussionDataChain: Prepare and curate datasets for AI/ML
2 points · Open discussionDataChain: Prepare and curate datasets for AI/ML
3 points · Open discussionDataChain: Prepare and curate datasets for AI/ML
2 points · Open discussionAn out-of-memory dataframe to wrangle unstructured data at scale - https://github.com/iterative/datachain
In Ask HN: What are you working on (August 2024)?DataChain: Dataframe to enrich, transform and analyze data for ML
1 points · Open discussionDataChain: A Pythonic data-frame library for artificial intelligence
2 points · Open discussionYou can use the https://github.com/iterative/datachain mentioned by @dmpetrov to predict and draw (in SaaS) a molecular structure. Not only can you predict, but you can also
- enrich the PDF data with external PDB data,
- calculate and evaluate sequence and structure-based predictions made by multiple custom models,
- and optimize time and resources.
I created some simple examples in this area a few months ago. Feel free to email me at mikhail@iterative.ai if you're interested in sharing my findings.
In Launch HN: Roe AI (YC W24) – AI-powered data warehouse to query multimodal dataBridging the gap between AI and data warehouses is crucial, but I’m not sure SQL is the best fit for AI engineers who mainly work with Python and AI APIs.
At DataChain, we are solving this by creating a Python API that translates to SQL under the hood, which is pretty easy now with Pydantic. https://github.com/iterative/datachain
WDYT?
In Launch HN: Roe AI (YC W24) – AI-powered data warehouse to query multimodal dataDataChain: Enrich, transform and curate datasets for ML
1 points · Open discussionDataChain: AI-dataframe to enrich, transform, analyze data from cloud storages
1 points · Open discussionShowing the 15 most recent of 17 sources. Enable JavaScript to search and browse every indexed mention.