rasbt

LLMs-from-scratch

Jupyter Notebook2024–2026

rasbt / LLMs-from-scratch

LLMs-from-scratch

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

AI & Machine Learning Learning & Reference

Explore on GitHub Read Hacker News discussions

22Distinct people

33Mentions

0Explicit recommenders

108.3KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year33 dated mentions

Observed mentions: 2024-01-27 16:19:42 UTC to 2026-10-26 21:55:12 UTC.

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	14	9	5
2025	2025-01-01 00:00:00 UTC	2026-01-01 00:00:00 UTC	14	8	6
2026 (partial)	2026-01-01 00:00:00 UTC	2027-01-01 00:00:00 UTC	5	1	4

What Hacker News says

33 sources

dtf·Oct 26, 2026

There's a note in Sebastian Raschka's LLMs-from-scratch github repository - "Recommendations for Getting the Most Out of a Technical Book". He recommends reading offline and code-free first (although chapter-by-chapter). https://github.com/rasbt/LLMs-from-scratch/blob/cdbd33e6d57a...

In Sebastian Raschka's from-scratch video course on building reasoning models

boredumb·Aug 24, 2026

I agree with mostly all of this, but personally I wrote a toy LLM almost 5 years ago and while it never saw much use outside of boring my wife with a shitty command line demo with glee it did help me understand how they worked and how to apply them, played a lot with JAX and pytorch, ended up building a ghetto version of MCP and an LLM-Pool to proxy requests to my baby local models and so I didn't struggle to see the evolution of openrouter and MCP agentic workflows. The same way i'm really glad when I was younger I built a bad webserver by myself, a really painful SQLx type database, etc etc etc - none of these things led me to developing for Nginx or Oracle nor will knowing JAX get me a job at an AI research lab, but I do have a lot of depth in understanding how the technology works so that the flavors on top of them are easy to digest

Read the rest

and make more use of immediately, and I think the same can be said for engineers coming into the field - if it's a spooky LLM box you aren't going to be squeezing the same amount of juice as the guy that knows how they work inside and out so having at least the understanding of a _babys first LLM_ is going to get you miles ahead of people who don't. For anyone who wants to dork around there is https://github.com/rasbt/LLMs-from-scratch which is something amazing that I think anyone who wants to engineer things around LLMs should at least blast through and read.

In I were 17, I'd learn how to build LLMs from scratch

eigenBasis·May 24, 2026Submission

DeepSeek Sparse Attention

2 points · Open discussion

JoeDaDude·May 5, 2026

Coincidentally, I just started on Build a Large Language Model (From Scratch), a repo/book/course by Sebastian Raschka [0][1][2]. Maybe it is a good problem to have to have to decide which learning resource to use. [0] https://github.com/rasbt/LLMs-from-scratch [1] https://www.manning.com/books/build-a-large-language-model-f... [2] https://magazine.sebastianraschka.com/p/coding-llms-from-the...

In Train Your Own LLM from Scratch

Propelloni·Apr 6, 2026

Great work! I still think that [1] does a better job of helping us understand how GPT and LLM work, but yours is funnier. Then, some criticism. I probably don't get it, but I think the HN headline does your project a disservice. Your project does not demystify anything (see below) and it diverges from your project's claim, too. Furthermore, I think you claim too much on your github. "This project exists to show that training your own language model is not magic." and then just posts a few command line statements to execute. Yeah, running a mail server is not magic, just apt-get install exim4. So, code. Looking at train_guppylm.ipynb and, oh, it's PyTorch again. I'm better off reading [2] if I'm looking into that (I know, it is a published book, but I maintain my point). So, in short, it does not help the initiated or the uninitiated. For

Read the rest

the initiated it needs more detail for it to be useful, the uninitiated more context for it to be understood. Still a fun project, even if oversold. [1] https://spreadsheets-are-all-you-need.ai/ [2] https://github.com/rasbt/LLMs-from-scratch

In Show HN: I built a tiny LLM to demystify how language models work

libraryofbabel·Dec 22, 2025

I read this article back when I was learning the basics of transformers; the visualizations were really helpful. Although in retrospect knowing how a transformer works wasn't very useful at all in my day job applying LLMs, except as a sort of deep background for reassurance that I had some idea of how the big black box producing the tokens was put together, and to give me the mathematical basis for things like context size limitations etc. I would strongly caution anyone who thinks that they will be able to understand or explain LLM behavior better by studying the architecture closely. That is a trap. Big SotA models these days exhibit so much nontrivial emergent phenomena (in part due to the massive application of reinforcement learning techniques) that give them capabilities very few people expected to ever see when this architecture fi

Read the rest

rst arrived. Most of us confidently claimed even back in 2023 that, based on LLM architecture and training algorithms, LLMs would never be able to perform well on novel coding or mathematics tasks. We were wrong. That points towards some caution and humility about using network architecture alone to reason about how LLMs work and what they can do. You'd really need to be able to poke at the weights inside a big SotA model to even begin to answer those kinds of questions, but unfortunately that's only really possible if you're a "mechanistic interpretability" researcher at one of the major labs. Regardless, this is a nice article, and this stuff is worth learning because it's interesting for its own sake! Right now I'm actually spending some vacation time implementing a transformer in PyTorch just to refresh my memory of it all. It's a lot of fun! If anyone else wants to get started with that I would highly recommend Sebastian Raschka's book and youtube videos as way into the subject: https://github.com/rasbt/LLMs-from-scratch . Has anyone read TFA author Jay Alammar's book (published Oct 2024) and would they recommend it for a more up-to-date picture?

In The Illustrated Transformer

quietlearning·Nov 24, 2025Submission

Olmo 3 from Scratch (Standalone Notebook)

1 points · Open discussion

ModelForge·Nov 3, 2025Submission

Explanation of Gated DeltaNet (Qwen3-Next and Kimi Linear)

3 points · Open discussion

malshe·Sep 6, 2025

Here is the code used in the book - https://github.com/rasbt/LLMs-from-scratch

In The maths you need to start understanding LLMs

ModelForge·Aug 20, 2025

Could be an artifact of the small size not fully taking advantage of the GPU. For example, for the slightly larger Qwen3 0.6B model the A100 is faster (you can see it when scrolling to the bottom here: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/11...)

In Gemma 3 270M re-implemented in pure PyTorch for local tinkering

ModelForge·Aug 20, 2025Submission

Gemma 3 270M re-implemented in pure PyTorch for local tinkering

417 points · Open discussion

tzury·Aug 2, 2025Submission

Qwen3 from Scratch

3 points · Open discussion

lufenialif2·Jul 2, 2025

I appreciate the effort! And I disagree - that's what it's all about haha I assume there are more levers we could try pulling to reduce variation? I'll be looking into this as well. As an aside, because of my own experience with variability using chatGPT (non-API, I assume there are also more levers to pull here), I've been thinking about LLMs and their application to gaming. To what extent it is possible to use LLMs to interpret a result and then return a variable that then executes the usual state updates? This would hopefully add a bit of intentional variability in the game's response to user inputs but consistency in updating internal game logic. edit: found this! https://github.com/rasbt/LLMs-from-scratch/issues/249 Seems that it's an ongoing issue from various other links I've found, and now when I google "ollama reproducibility" th

Read the rest

is thread comes up on the first page, so it seems it's an uncommon issue as well :(

In LLMs Bring New Nature of Abstraction

sbbq·Jun 20, 2025Submission

Qwen3 Implemented from Scratch

3 points · Open discussion

jinqueeny·Apr 24, 2025Submission

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

2 points · Open discussion

Showing the 15 most recent of 33 sources. Enable JavaScript to search and browse every indexed mention.