HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
karpathy
nanoGPT
Python2023–2026
karpathy / nanoGPT

nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

48Distinct people
49Mentions
0Explicit recommenders
64.7KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

The simplest, fastest repository for training/finetuning medium-sized GPTs.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 5:58:53 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year49 dated mentions

Observed mentions: 2023-01-03 18:05:25 UTC to 2026-09-30 01:12:04 UTC.

05101520252023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 25 mentions; 5 submissions, 20 comments20232024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 13 mentions; 3 submissions, 10 comments20242025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 10 mentions; 1 submissions, 9 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC25520
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC13310
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC1019
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC101

What Hacker News says

13 sources
You would have to train the new model from scratch since it would be all new token embeddings with whatever character encoding scheme you come up with. It would probably make sense to train the vanilla gpt from scratch with the same total embeddings size as your control. I would start with https://github.com/karpathy/nanoGPT as a baseline since you can train a toy (GPT2 sized) llm in a couple days on an a100 which are pretty easy to come by.
In Probably pay attention to tokenizers
There are a bunch of good answers, but I wanted to succinctly say "practically, quite a bit". Here's a good little rabbit-hole example: > https://github.com/karpathy/nanoGPT/blob/master/model.py#L45 Karpathy's nanoGPT calling flash attention by checking if torch.nn.functional.scaled_dot_product_attention exists > https://pytorch.org/docs/stable/generated/torch.nn.functiona... Looking at the docs, in reality, most of the time you want this to call out to FA2 which optimizes the kernals on the device to split ops on the Softmax of the triangular matrix as well as reduce moving unnecessary batches of floating point numbers back and forth from the GPU to the CPU. > https://arxiv.org/pdf/2307.08691 The paper for FA2 almost entirely considers itself through the hardware it's running on.
In FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision
For what it's worth, the idea of a "tensor" in ML is pretty far removed from any physical concept. I don't know its mathematical origins (would be interesting I'm sure), but in ML they're only involved because that's our framework for dealing with multi-linear transformations. Most NNs work by something akin to "(multi-)linear vector transformation, followed by elementwise nonlinear transformation", stacked over and over so that the output of one layer becomes the input of the next. This applies equally well to simple models like "fully-connected" / "feed-forward" networks (aka "multi-layer perceptron") and to more-sophisticated models like transformers (e.g. https://github.com/karpathy/nanoGPT/blob/325be85d9be8c81b436...). It's less about combining lots of tiny local linear transformations piecewise, and more about layering linear and no
Read the rest
n-linear transformations on top of each other. I don't really know how physics works beyond whatever Newtonian mechanics I learned in high school. But unless the underlying math is similar, then I'm hesitant to run too far with the analogy.
In GPT-4o
Nice work! I built something similar years ago and I did compile the probabilities based on a corpus of text (public domain books) in an attempt to produce writing in the style of various authors. The results were actually quite similar to the output of nanoGPT[0]. It was very unoptimized and everything was kept in memory. I also knew nothing about embeddings at the time and only a little about NLP techniques that would certainly have helped. Using a graph database would have probably been better than the datastructure I came up with at the time. You should look into stuff like Datalog, Tries[1], and N-Triples[2] for more inspiration. Your idea of splitting the probabilities based on whether you're starting the sentence or finishing it is interesting but you might be able to benefit from an approach that creates a "window" of text you can
Read the rest
use for lookup, using an LCS[3] algorithm could do that. There's probably a lot of optimization you could do based on the probabilities of different sequences, I think this was the fundamental thing I was exploring in my project. Seeing this has inspired me further to consider working on that project again at some point. [0] https://github.com/karpathy/nanoGPT [1] https://en.wikipedia.org/wiki/Trie [2] https://en.wikipedia.org/wiki/N-Triples [3] https://en.wikipedia.org/wiki/Longest_common_subsequence
In Show HN: Predictive text using only 13kb of JavaScript. no LLM
metalwhale·Feb 4, 2024Submission

Ask HN: How LLMs Learn to Be "Generative"?

AFAIK to be called "generative", a model should have the ability to learn the joint probability over the training data. In the case of LLMs, we apply the chain rule of Bayes' formula to achieve this by leveraging the autoregressive method for every token of each input text sequence. For example, with a text sequence of 4 tokens, it can be written as: p(x4,x3,x2,x1) = p(x4|x3,x2,x1) * p(x3|x2,x1) * p(x2|x1) * p(x1) where x1 denotes the 1st token, x2 denotes the 2nd token and so on, respectively. I understand the conditional terms p(x_n|...) where we use cross-entropy to calculate their losses. However, I'm unsure about the probability of the very first token p(x1). How is it calculated? Is it in some configurations of the training process, or in the model architecture, or in the loss function? IMHO, if the model doesn't learn p(x1) properl
Read the rest
y, the entire formula for Bayes' rule cannot be completed, and we can't refer to LLMs as "truly generative". Am I missing something here? I asked the same question on nanoGPT repo: https://github.com/karpathy/nanoGPT/issues/432, but I haven't found the answer I'm looking for yet. Could someone please enlighten me? Thank in advance!
2 points · Open discussion
For training from scratch, maybe a small model like https://github.com/karpathy/nanoGPT or tinyllama. Perhaps with quantization. Fine-tuning is very doable. The hard part is making a novel dataset with input output pairs. You might consider just combining datasets you find on HuggingFace as an experiment. replicate.com has a dead simple fine tuning API. Predibase is also an easy to use option. But again for something custom you need a dataset with hundreds of examples. Normally people use GPT-4 to generate the dataset. As long as OpenAI doesn't block them.
In Ask HN: Is it feasible to train my own LLM?

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.