SCScene_Cast2·Feb 5, 2025
I'm curious, what does paradigmatic JAX look like? Is there an equivalent of picoGPT [1] for JAX?
[1] https://github.com/jaymody/picoGPT/blob/main/gpt2.py
In How to scale your model: A systems view of LLMs on TPUsjaymody
picoGPT
Python2023–2025
jaymody / picoGPT
picoGPT
An unnecessarily tiny implementation of GPT-2 in NumPy.
4Distinct people
5Mentions
0Explicit recommenders
3.5KGitHub stars
What to know before exploring
What it is
An unnecessarily tiny implementation of GPT-2 in NumPy.
GitHub
Where it is used
No documented product use has been verified yet.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by month5 dated mentions
Observed mentions: 2023-02-23 11:33:55 UTC to 2025-02-05 01:36:21 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Feb 2023 | 2023-02-01 00:00:00 UTC | 2023-03-01 00:00:00 UTC | 1 | 1 | 0 |
| Mar 2023 | 2023-03-01 00:00:00 UTC | 2023-04-01 00:00:00 UTC | 1 | 0 | 1 |
| Apr 2023 | 2023-04-01 00:00:00 UTC | 2023-05-01 00:00:00 UTC | 1 | 0 | 1 |
| May 2023 | 2023-05-01 00:00:00 UTC | 2023-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2023 | 2023-06-01 00:00:00 UTC | 2023-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2023 | 2023-07-01 00:00:00 UTC | 2023-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2023 | 2023-08-01 00:00:00 UTC | 2023-09-01 00:00:00 UTC | 1 | 0 | 1 |
| Sep 2023 | 2023-09-01 00:00:00 UTC | 2023-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2023 | 2023-10-01 00:00:00 UTC | 2023-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2023 | 2023-11-01 00:00:00 UTC | 2023-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2023 | 2023-12-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2024 | 2024-01-01 00:00:00 UTC | 2024-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2024 | 2024-02-01 00:00:00 UTC | 2024-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2024 | 2024-03-01 00:00:00 UTC | 2024-04-01 00:00:00 UTC | 0 | 0 | 0 |
| Apr 2024 | 2024-04-01 00:00:00 UTC | 2024-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2024 | 2024-05-01 00:00:00 UTC | 2024-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2024 | 2024-06-01 00:00:00 UTC | 2024-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2024 | 2024-07-01 00:00:00 UTC | 2024-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2024 | 2024-08-01 00:00:00 UTC | 2024-09-01 00:00:00 UTC | 0 | 0 | 0 |
| Sep 2024 | 2024-09-01 00:00:00 UTC | 2024-10-01 00:00:00 UTC | 0 | 0 | 0 |
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 0 | 0 | 0 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 0 | 0 | 0 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 1 | 0 | 1 |
What Hacker News says
5 sourcesIn that case, you might also enjoy https://jaykmody.com/blog/gpt-from-scratch/
(here's the raw code: https://github.com/jaymody/picoGPT/blob/main/gpt2.py)
In Understanding Automatic Differentiation in 30 lines of PythonI wrote a minimal implementation in NumPy here (the forward pass code is only 40 lines): https://github.com/jaymody/picoGPT
And also a related blog post: https://news.ycombinator.com/item?id=34726115
Although this is for a decoder-only transformer (aka GPT) and doesnt include the encoder part.
In Transformers from Scratch (2021)Take a look at https://github.com/jaymody/picoGPT/blob/a750c145ba4d09d57648...
Yes, this is GPT-2 not 4 and it‘s not the RL-trained Chat, only the GPT model and it‘s basically only the inference part, not the training loop and it‘s somewhat simplified.
Still, take a good look.
That‘s essentially what it is on a single sheet of paper.
There is nothing specifically about language in „language model“, we just call it that. Better to call it just LLM.
Nobody knows exactly what it learns, although there would be ways to poke around given some research programs. But it seems like the interest in that is limited currently, everyone is busy with improving it or with applications.
Perhaps the answer is that we overestimated what a mind is. It‘s like we used to ask what life is and it turned out that there is nothing special about life, not even th
Read the rest
e DNA is controlling anything. It‘s merely a chemical process, even though a complex process.
Personally, I believe there is something special about consciousness and I believe these systems are not conscious. And I believe they are currently not flexible enough to adapt to new circumstances, unlike a human brain. They don‘t currently make much real progress. But that may change.
PicoGPT: An unnecessarily tiny implementation of GPT-2 in NumPy
3 points · Open discussion