data_engineering_book

I'm not sure whether this is an artefact of translation, but things like this don't inspire confidence: > The "Modern Data Stack" (MDS) is a hot concept in data engineering in recent years, referring to a cloud-native, modular, decoupled combination of data infrastructure https://github.com/datascale-ai/data_engineering_book/blob/m... Later parts are better and more to the point though: https://github.com/datascale-ai/data_engineering_book/blob/m... Edit: perhaps I judged to early. The RAG sections isn't bad either: https://github.com/datascale-ai/data_engineering_book/blob/m...

Oh thanks! I've switched the top URL to that now. Submitted URL was https://github.com/datascale-ai/data_engineering_book. I hope xx123122 won't mind my mentioning that they emailed us about this post, which originally got caught in a spam filter. I invited them to post a comment giving the background to the project but they probably haven't seen my reply yet. Hopefully soon, given that the post struck a chord! Edit: they did, and I've moved that post to the toptext.

English version: https://github.com/datascale-ai/data_engineering_book/blob/m...

Hi HN! I'm currently a Master's student at USTC (University of Science and Technology of China). I've been diving deep into Data Engineering, especially in the context of Large Language Models (LLMs). The Problem: I found that learning resources for modern data engineering are often fragmented and scattered across hundreds of medium articles or disjointed tutorials. It's hard to piece everything together into a coherent system. The Solution: I decided to open-source my learning notes and build them into a structured book. My goal is to help developers fast-track their learning curve. Key Features: LLM-Centric: Focuses on data pipelines specifically designed for LLM training and RAG systems. Scenario-Based: Instead of just listing tools, I compare different methods/architectures based on specific business scenarios (e.g., "When to use Vect