curator

I keep seeing synthetic data pipelines powering the latest LLM “breakthroughs”: • TinyZero’s $30 fine-tuning workflow • Sky-T1’s $450 reasoning-model build • Meta AI’s Llama 3 herd (2024 paper detailing their synthetic-data training) • Berkeley OpenThoughts (“Data Recipes for Reasoning Models”), published yesterday There are also open-source toolkits you can experiment with: https://github.com/meta-llama/synthetic-data-kit https://github.com/bespokelabsai/curator But it still feels very research-oriented. I haven’t found many examples of these pipelines running in real-world products. I’m curious: 1. Who is using synthetic-data pipelines in production today? 2. What tasks does it actually improve. E.g. fine-tuning smaller models for specific tasks? Any real-world stories, pointers, or further reading would be hugely appreciated. Thanks!

Synthetic data generation is an essential step in training and evaluating LLMs/Agents/RAG pipelines, but tooling around this is still lacking. We're introducing Curator, an open-source library designed to streamline the data curation process. While there are many libraries to prompt LLMs, the semantics of generating synthetic data is different from prompting. For example, we need to process a large number of prompts (sometimes in millions or more) while accepting some failures, utilize several stages of prompting, incorporate human feedback, and filter out bad data using verifiers and heuristics. Curator addresses these challenges: 1. It supports efficient data generation by several API providers and local models. 2. Recovers from failures and caches previous output. 3. Utilizes structured outputs to enable programming complex data genera