tysam-code

hlb-CIFAR10

Python2022–2025

tysam-code / hlb-CIFAR10

hlb-CIFAR10

Train to 94% on CIFAR-10 in <6.3 seconds on a single A100. Or ~95.79% in ~110 seconds (or less!)

Explore on GitHub Read Hacker News discussions

8Distinct people

23Mentions

1Explicit recommenders

1.3KGitHub stars

REPOSITORY AT A GLANCE

What to know before exploring

What it is

Train to 94% on CIFAR-10 in <6.3 seconds on a single A100. Or ~95.79% in ~110 seconds (or less!)

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

I really love small neural networks. They have some nice properties that people overlook. The training speed record (warning, self promo) for CIFAR10 to 94% uses a very tiny…

tysam_and used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

kellerjordan/modded-nanogpt · linked in 2 of the same discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 12:01:02 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year23 dated mentions

Observed mentions: 2022-12-12 01:38:51 UTC to 2025-09-06 14:17:09 UTC.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges

UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
Period	Start (UTC)	End (exclusive, UTC)	Mentions	Submissions	Comments
2022	2022-01-01 00:00:00 UTC	2023-01-01 00:00:00 UTC	3	1	2
2023	2023-01-01 00:00:00 UTC	2024-01-01 00:00:00 UTC	16	3	13
2024	2024-01-01 00:00:00 UTC	2025-01-01 00:00:00 UTC	2	0	2
2025	2025-01-01 00:00:00 UTC	2026-01-01 00:00:00 UTC	2	0	2

What Hacker News says

3 sources

tbalsam·Dec 23, 2022

Someone else linked this in a different tree but I do truly love this blogpost series -- you can see the section "input patch whitening" for more details. https://myrtle.ai/learn/how-to-train-your-resnet-8-bag-of-tr... Alternatively, you can read through the direct code here if you're willing to wade through some of the mathy code on it: https://github.com/tysam-code/hlb-CIFAR10/blob/main/main.py#...

In Do simpler machine learning models exist and how can we find them?

tbalsam·Dec 22, 2022

I recently released a codebase in beta that modernizes a tiny model that gets really good performance on CIFAR-10 in about 18.1 or so seconds on the right single GPU -- a number of years ago the world record was 10 minutes, down from several days a few years previously. While most of my work was porting and cleaning up certain parts of the code for a different purpose (just-clone-and-hack experimentation workbench), I've spent years optimizing neural networks at a very fine grained level, and many of the lessons learned here in debugging reflected that. I believe that there are fundamentally a few big NP-hard layers (at least two that I can define, and likely several other smaller ones) unfortunately but they are not hard blockers to progress. The model I mentioned above is extremely simple and has little "extra fat" where it is not neede

Read the rest

d. It also importantly seems to have good gradient and such flow throughout, something that's important for a model to be able to learn quickly. There are a few reasonable priors, like initializing and freezing the first convolution to whiten the inputs based upon some statistics from the training data. That does a shocking amount of work in stabilizing and speeding up training. Ultimately, the network is simple, and there are a number of other methods to help it reach near-SOTA, but they are as simple as can be. I think as this project evolves and we get nearer to the goal (<2 seconds in a year or two), we'll keep uncovering good puzzle pieces showing exactly what it is that's allowing such a tiny network to perform so well. There's a kind of exponential value to having ultra-short training times -- you can somewhat open-endedly barrage-test your algorithm, something that's already led to a few interesting discoveries that I'd like to refine before publishing to the repo. If you're interested, the code is here. The running code is a single .py with the upsides and downsides that come with that. If you're interested or have any questions, let me know! :D :)))) https://github.com/tysam-code/hlb-CIFAR10

In Do simpler machine learning models exist and how can we find them?

tysam_and·Dec 12, 2022Submission

Show HN: Hlb-CIFAR10 – 94% accuracy in 18.1 seconds, A100, ground-up hackable

Quick Start To get started (if you've done your code scan due diligence and you're setup on CUDA), you can just run `git clone https://github.com/tysam-code/hlb-CIFAR10 && cd hlb-CIFAR10 && python -m pip install -r requirements.txt && python main.py` and you should see things training straightaway. If you have a good internet connection and card, your first run should be done within 90 seconds (including dependency installs, dataset download, etc). Project Introduction Hi there, I've been working in the modern instantiation of this field for just over half a decade or so, and have always sort of wanted a good testbench to develop neural network research on. Most of the codebases I've worked with over the years have been very ad-hoc or not very close to the bleeding edge. Or, the highly performant codebases really weren't amenable to hacki

Read the rest

ng -- just for performing the main function they were built for (a single paper, benchmark, service, etc). I've spent 120-130 hours or so rebuilding David Page's lovely record-setting dawnbench project (https://myrtle.ai/learn/how-to-train-your-resnet/) in a way that is generic and amenable to both hacking and rapid experimentation/prototyping. One benefit of this is that the original code is extremely performance optimized, so not only is it an easy workbench to prototype on, but a good testbed for if a new method is orthogonal or not to older methods. I've tried to make it as absolutely plug and play as possible, and I hope this offers a nice combination of being fun and useful. I've already gotten a significant amount of value in testing some methods that would otherwise be very hard to test with it. In both cases, either change required just a single line to be added, which I felt validated the original design decisions. I have some hopefully-not-to-nefarious plans for it if I can get it down to the ~2 seconds mark within a few years. I have a Patreon for those interested in seeing more similar work from me in the future. You can get that and other info from the github link above. I plan on hanging around in the comments for a while in case anyone has any questions. Please do let me know, I love the conversation, and whether or not you like/liked it, or if you find this intriguing or not -- this helps me build better software in the future! So please, feel free to drop a line. I'd love to hear what you think! :) :D

1 points · Open discussion