JUJunnn·Mar 17, 2026Submission
Show HN: DataFlow,Turn raw data into high-quality LLM training datasets
2 points · Open discussionOpenDCAI
DataFlow
Python2025–2026
OpenDCAI / DataFlow
DataFlow
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
3Distinct people
4Mentions
0Explicit recommenders
8.6KGitHub stars
What to know before exploring
What it is
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 12:54:37 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month4 dated mentions
Observed mentions: 2025-07-14 15:03:13 UTC to 2026-03-17 05:38:59 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 1 | 1 | 0 |
| Aug 2025 | 2025-08-01 00:00:00 UTC | 2025-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2025 | 2025-09-01 00:00:00 UTC | 2025-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2025 | 2025-10-01 00:00:00 UTC | 2025-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2025 | 2025-11-01 00:00:00 UTC | 2025-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2025 | 2025-12-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 1 | 1 | 0 |
| Jan 2026 | 2026-01-01 00:00:00 UTC | 2026-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2026 | 2026-02-01 00:00:00 UTC | 2026-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2026 | 2026-03-01 00:00:00 UTC | 2026-04-01 00:00:00 UTC | 2 | 2 | 0 |
What Hacker News says
4 sourcesShow HN: Generate, Clean, and Prepare LLM Training Data, All-in-One
2 points · Open discussionShow HN: DataFlow – Open Tool for LLM data prep 10k synthetic > 1M generic data
Hi HN,
We are the OpenDCAI team from Peking University. We just released DataFlow, an open-source framework designed to make LLM data preparation as programmable and modular as model training.
The Problem: While model architectures are standardized (PyTorch/JAX), data preparation is still dominated by ad-hoc scripts and loosely defined workflows. Most existing tools focus on "cleaning" or "filtering" existing large datasets, but modern LLM training increasingly relies on complex synthetic data generation and iterative refinement.
Our Solution: DataFlow DataFlow treats data processing like constructing a neural network. It provides a PyTorch-like programming interface where you compose Operators into Pipelines.
Key Technical Features:
- Modular Abstraction: Just like torch.nn.Module, we provide standard interfaces for Operators, Prompt Te
Read the rest
mplates, and Pipelines.
- Rich Operator Zoo: Nearly 200 pre-built operators covering Text, Math, Code, Text-to-SQL, and RAG.
- DataFlow-Agent: An agentic layer (built on LangGraph) that translates natural language requirements directly into executable pipelines.
Results: We found that data quality matters more than scale.
- 10k > 1M: A unified 10k-sample dataset produced by DataFlow enables base models (Qwen2/2.5) to surpass counterparts trained on 1M generic instruction samples (Infinity-Instruct).
- Code & SQL: Our pipelines achieved +7% improvement on code benchmarks and +3% execution accuracy in Text-to-SQL using significantly less data.
Links:
- Paper: https://arxiv.org/abs/2512.16676
- Repo: https://github.com/OpenDCAI/DataFlow
- Docs:https://opendcai.github.io/DataFlow-Doc/
We believe data engineering deserves the same level of rigorous abstraction as model architecture. We hope DataFlow can serve as a foundational substrate for future data-centric AI development.
Show HN: DataFlow: makes LLM data processing fast, powerful, and EASY
3 points · Open discussion