The discussion centers on Microsoft Research's disbandment of its AI Frontiers group, with participants debating whether this reflects a rational industry pivot toward scaling transformers or a destructive abandonment of fundamental research. Strong disagreements emerge regarding the validity of scaling laws, the competence of leadership decisions like firing Yann LeCun, and the broader health of corporate R&D culture.
236 readable replies · Supporting reply 1 · Supporting reply 2 · Supporting reply 3 · Supporting reply 4KellerJordan
modded-nanogpt
Python2024–2026
KellerJordan / modded-nanogpt
modded-nanogpt
NanoGPT (124M) in 90 seconds
16Distinct people
21Mentions
1Explicit recommenders
6KGitHub stars
What to know before exploring
What it is
NanoGPT (124M) in 90 seconds
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
> Performance of BDH-GPU and GPTXL versus model size on a translation task. [...] On the other hand, GPTXL [...] required Dropout [...] The model architecture follows GPT2 I…
kouteiheika used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- tysam-code/hlb-cifar10 · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 5:04:57 PM UTC.
Mentions over time
HN mentions by month21 dated mentions
Observed mentions: 2024-10-12 20:29:05 UTC to 2026-09-29 01:47:21 UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Oct 2024 | 2024-10-01 00:00:00 UTC | 2024-11-01 00:00:00 UTC | 1 | 1 | 0 |
| Nov 2024 | 2024-11-01 00:00:00 UTC | 2024-12-01 00:00:00 UTC | 0 | 0 | 0 |
| Dec 2024 | 2024-12-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2025 | 2025-01-01 00:00:00 UTC | 2025-02-01 00:00:00 UTC | 2 | 0 | 2 |
| Feb 2025 | 2025-02-01 00:00:00 UTC | 2025-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2025 | 2025-03-01 00:00:00 UTC | 2025-04-01 00:00:00 UTC | 1 | 0 | 1 |
| Apr 2025 | 2025-04-01 00:00:00 UTC | 2025-05-01 00:00:00 UTC | 0 | 0 | 0 |
| May 2025 | 2025-05-01 00:00:00 UTC | 2025-06-01 00:00:00 UTC | 1 | 0 | 1 |
| Jun 2025 | 2025-06-01 00:00:00 UTC | 2025-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2025 | 2025-07-01 00:00:00 UTC | 2025-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2025 | 2025-08-01 00:00:00 UTC | 2025-09-01 00:00:00 UTC | 4 | 0 | 4 |
| Sep 2025 | 2025-09-01 00:00:00 UTC | 2025-10-01 00:00:00 UTC | 1 | 0 | 1 |
| Oct 2025 | 2025-10-01 00:00:00 UTC | 2025-11-01 00:00:00 UTC | 3 | 0 | 3 |
| Nov 2025 | 2025-11-01 00:00:00 UTC | 2025-12-01 00:00:00 UTC | 1 | 0 | 1 |
| Dec 2025 | 2025-12-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 0 | 0 | 0 |
| Jan 2026 | 2026-01-01 00:00:00 UTC | 2026-02-01 00:00:00 UTC | 2 | 0 | 2 |
| Feb 2026 | 2026-02-01 00:00:00 UTC | 2026-03-01 00:00:00 UTC | 0 | 0 | 0 |
| Mar 2026 | 2026-03-01 00:00:00 UTC | 2026-04-01 00:00:00 UTC | 1 | 1 | 0 |
| Apr 2026 | 2026-04-01 00:00:00 UTC | 2026-05-01 00:00:00 UTC | 1 | 0 | 1 |
| May 2026 | 2026-05-01 00:00:00 UTC | 2026-06-01 00:00:00 UTC | 0 | 0 | 0 |
| Jun 2026 | 2026-06-01 00:00:00 UTC | 2026-07-01 00:00:00 UTC | 0 | 0 | 0 |
| Jul 2026 | 2026-07-01 00:00:00 UTC | 2026-08-01 00:00:00 UTC | 0 | 0 | 0 |
| Aug 2026 | 2026-08-01 00:00:00 UTC | 2026-09-01 00:00:00 UTC | 2 | 1 | 1 |
| Sep 2026 | 2026-09-01 00:00:00 UTC | 2026-10-01 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
21 sourcesSIsimonpure·Sep 29, 2026Submission
Training NanoGPT in 39.9 Seconds
3 points · Open discussionI misread the title and thought it would be about the (for lack of a better term) NanoGPT speedrun[1]. Which, previous to the article, was meant to be the world speed records for Andrej Karpathy's GPT-2 (small) reproduction.
1. https://github.com/KellerJordan/modded-nanogpt#world-record-...
In NanoGPT Speedrun FrontierIt's also a great base for experimentation. If you have an idea for an architecture improvement you can try it for $36 on the 20 layer nanocode setting, then for another $200 see how it holds up on the "full scale" nanocode
Kaparthy's notes on improving nanochat [1] are one of my favorite blog-like things to read. Really neat to see which features have how much influence, and how the scaling laws evolve as you improve the architecture
There's also modded-nanogpt which turns the same kind of experimentation into a training speedrun (and maybe loses some rigor on the way) [2]
1 https://github.com/karpathy/nanochat/blob/master/dev/LOG.md
2 https://github.com/kellerjordan/modded-nanogpt
In Nanocode: The best Claude Code that $200 can buy in pure JAX on TPUsmodded-nanogpt: NanoGPT (124M) in 2 minutes
2 points · Open discussionCool post. And you might find it interesting. https://github.com/KellerJordan/modded-nanogpt
In Training a 67M-parameter transformer on an M4 Mac MiniI'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems.
https://github.com/KellerJordan/modded-nanogpt
In Starting from scratch: Training a 30M Topological TransformerWhen it comes to compute cost the choice of activation function makes little difference nowadays (and it can often be fused with whatever operation comes before it, which makes it effectively free).
The real reason is simple: it was inherited.
The relu^2 was used in the nanogpt speedrun[1] because it produced the best empirical results, then Andrej based his nanochat on the nanogpt speedrun without changing the activation function, and then this project was based on nanochat.
[1] -- https://github.com/KellerJordan/modded-nanogpt
In Show HN: Tiny Diffusion – A character-level text diffusion model from scratchScaling applies to multiple dimensions simultaneously over time. A frontier model today could be replicated a year later with a model half the size, with a quarter of the FLOPS, etc. I don’t know the real numbers for optimization scaling, but you could check out NanoGPT speedrun [1] as an example.
The best solution in the meantime is giving the LLM a harness that allows tool use like what coding agents have. I suspect current models are fully capable of solving arbitrary complexity artificial reasoning problems here, provided that they’re used in the context of a coding agent tool.
[1] https://github.com/KellerJordan/modded-nanogpt
In Reasoning models reason well, until they don't> Performance of BDH-GPU and GPTXL versus model size on a translation task. [...] On the other hand, GPTXL [...] required Dropout [...] The model architecture follows GPT2
I love when a new architecture comes out, but come on, it's 2025, can we please stop comparing fancy new architectures to the antiquated GPT2? This makes the comparison practically, well, useless. Please pick something more modern! Even the at-this-point ubiquitous Llama would be a lot better. I don't want to have to spend days of my time doing my own benchmarks to see how it actually compares to a modern transformer (and realistically, I was burned so many times now that I just stopped bothering).
Modern LLMs are very similar to GPT2, but those architectural tweaks do matter and can make a big difference. For example, take a look at the NanoGPT speedrun[1] and look at
Read the rest
how many training speedups they got by tweaking the architecture.
Honestly, everyone who publishes a paper in this space should read [2]. The post talks about optimizers, but this is also relevant to new architectures too. Here's the relevant quote:
> With billions of dollars being spent on neural network training by an industry hungry for ways to reduce that cost, we can infer that the fault lies with the research community rather than the potential adopters. That is, something is going wrong with the research. Upon close inspection of individual papers, one finds that the most common culprit is bad baselines [...]
> I would like to note that the publication of new methods which claim huge improvements but fail to replicate / live up to the hype is not a victimless crime, because it wastes the time, money, and morale of a large number of individual researchers and small labs who run and are disappointed by failed attempts to replicate and build on such methods every day.
Sure, a brand new architecture is most likely not going to compare favorably to a state-of-art transformer. That is fine! But at least it will make the comparison actually useful.
[1] -- https://github.com/KellerJordan/modded-nanogpt
[2] -- https://kellerjordan.github.io/posts/muon/#discussion-solvin...
> nanochat is also inspired by modded-nanoGPT
Nice synergy here, the lineage is: Karpathy's nano-GPT -> Keller Jordan's modded-nanoGPT (a speedrun of training nanoGPT) -> NanoChat
modded-nanoGPT [1] is a great project, well worth checking out, it's all about massively speeding up the training of a small GPT model.
Notably it uses the author's Muon optimizer [2], rather than AdamW, (for the linear layers).
[1] https://github.com/KellerJordan/modded-nanogpt
[2] https://kellerjordan.github.io/posts/muon/
In NanoChat – The best ChatGPT that $100 can buyLocation: MD, USA
Remote: Yes, open to hybrid/in-person for the right opportunity
Willing to relocate: Yep
Technologies: ML stack tech – PyTorch, Python, some Docker, Transformers, ConvNets, LLMs, etc
Résumé/CV: https://gist.githubusercontent.com/tysam-code/a7c49dcb72416b9cf62c399f882ae7b5/raw… (curlable, originally made for 80 px terminal. basic tech filter)
Email: hi [.dot.] re [.dot.] tysam [atsymbol] gmail [.dot.] com
Hi! I’m fern. I’m an experienced ML researcher and developer (almost 10 years total!) with a wide arena of experience including computer vision and language modeling. I have an excellent intuition for research directions and am good at building codebases that allow for rapid research iteration. In the open source world, I’ve set the world records for seve
Read the rest
ral benchmarks, nearly tripling the speed of a longstanding CIFAR10 world record (~17.1s -> 6.3s, https://github.com/tysam-code/hlb-CIFAR10, recognized by Karpathy: https://x.com/karpathy/status/1620103412686942208), setting a modded-nanogpt speed record and helping with several others (5.03m -> 4.66m, https://github.com/KellerJordan/modded-nanogpt/blob/master/r... ), improving the relative loss of DiLoCo speedrunning on modded-nanogpt by over 40% (https://x.com/hi_tysam/status/1928561266533990587), and miscellaneous other misadventures throughout the years.
I’m looking to be challenged in a strong technical environment with other people who I can learn from, and I’m open to work on most interesting problems that I think are neutral or bring something positive to the world. I’m very hard-problem oriented, and am quite skilled at distilling complex, slow, brittle training procedures down to simple, performant, and hackable ones. The kinds of problems that engage me the most are ones where I can always be growing and learning, and that’s the kind of environment that I’ll thrive in. Learning new subdomains (molecular, etc) is something that keeps me driven, I love the process of expanding out my toolkit. I also very much enjoy working on problems well within my domain of expertise.
My focus is on data/architecture/training dynamics and debugging for training efficiency more than raw low-level kernel creation, though the two go hand-in-hand. I enjoy mentoring people quite a bit! I am looking to learn working with larger-scale distributed runs, scale is something that I’d like to work with a bit more.
If you think this may align with what you’re doing – say hi! I’d love to chat.
Plenty of research involves small models trained on small amounts of data. You don't necessarily need to do an internet-scale training run to test a new model architecture, you can just compare it to other models of the same size trained on the same data. For example, small-model speedruns are a thing: https://github.com/KellerJordan/modded-nanogpt
In From multi-head to latent attention: The evolution of attention mechanismshttps://github.com/KellerJordan/modded-nanogpt is pretty great in that respect
In What's the strongest AI model you can train on a laptop in five minutes?I suspect one can go a lot further by adopting some tweaks from the GPT-2 speedrun effort [0], at minimum Muon, better init and carefully tuning learning rate.
[0]: https://github.com/KellerJordan/modded-nanogpt
In What's the strongest AI model you can train on a laptop in five minutes?Showing the 15 most recent of 21 sources. Enable JavaScript to search and browse every indexed mention.