HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
huggingface
transformers
Python2018–2026
huggingface / transformers

transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

AI & Machine LearningCreative & MediaLearning & Reference
Explore on GitHubRead Hacker News discussions
79Distinct people
93Mentions
1Explicit recommenders
168.9KGitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

GitHub

Where it is used

No documented product use was verified in the last source check.

What people are saying

For deep learning / neural networks, I think these are the best balance of theory and practice (not to mention great visualizations): http://cs231n.github.io More on the…

jointpdf used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

Mentioned in the same Hacker News threads. That is not proof they are substitutes.

This week on Hacker News

Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 10:56:20 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by year93 dated mentions

Observed mentions: 2018-11-07 08:59:39 UTC to 2026-08-06 23:03:10 UTC.

051015202018-01-01 00:00:00 UTC to 2019-01-01 00:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 comments20182019-01-01 00:00:00 UTC to 2020-01-01 00:00:00 UTC (exclusive): 9 mentions; 3 submissions, 6 comments2020-01-01 00:00:00 UTC to 2021-01-01 00:00:00 UTC (exclusive): 15 mentions; 3 submissions, 12 comments2021-01-01 00:00:00 UTC to 2022-01-01 00:00:00 UTC (exclusive): 7 mentions; 1 submissions, 6 comments2022-01-01 00:00:00 UTC to 2023-01-01 00:00:00 UTC (exclusive): 8 mentions; 1 submissions, 7 comments20222023-01-01 00:00:00 UTC to 2024-01-01 00:00:00 UTC (exclusive): 20 mentions; 4 submissions, 16 comments2024-01-01 00:00:00 UTC to 2025-01-01 00:00:00 UTC (exclusive): 12 mentions; 2 submissions, 10 comments2025-01-01 00:00:00 UTC to 2026-01-01 00:00:00 UTC (exclusive): 12 mentions; 3 submissions, 9 comments2026-01-01 00:00:00 UTC to 2027-01-01 00:00:00 UTC (exclusive): 9 mentions; 2 submissions, 7 comments. Incomplete calendar period.2026*

* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
20182018-01-01 00:00:00 UTC2019-01-01 00:00:00 UTC110
20192019-01-01 00:00:00 UTC2020-01-01 00:00:00 UTC936
20202020-01-01 00:00:00 UTC2021-01-01 00:00:00 UTC15312
20212021-01-01 00:00:00 UTC2022-01-01 00:00:00 UTC716
20222022-01-01 00:00:00 UTC2023-01-01 00:00:00 UTC817
20232023-01-01 00:00:00 UTC2024-01-01 00:00:00 UTC20416
20242024-01-01 00:00:00 UTC2025-01-01 00:00:00 UTC12210
20252025-01-01 00:00:00 UTC2026-01-01 00:00:00 UTC1239
2026 (partial)2026-01-01 00:00:00 UTC2027-01-01 00:00:00 UTC927

What Hacker News says

7 sources
lvwerra·Dec 10, 2021
We are releasing CodeParrot - my first project at Hugging Face! What is it? It is a code base to build large language models for code generation from scratch. It includes the code to clean up GitHub scale datasets, train GPT-2 style models from scratch on distributed infrastructure, and an evaluation benchmark of OpenAI's HumanEval dataset. We are also releasing two trained models and some demos to play with the models: https://huggingface.co/spaces/lvwerra/codeparrot-generation This is my first project at Hugging Face and I thought this is a good opportunity to talk about some of the difficulties behind the scenes of such a project, which are usually not so much communicated after its release. The topic of the story: distributed debugging is hard! The first few small experiments training GPT-2 models on a code dataset went well, so we d
Read the rest
ecided to scale and train the first smallish model for longer. For mysterious reasons, the training just stopped after a few hours. Not with an error, but the training loop just didn't continue. Even more interesting, when repeating the experiment it happened again but always at another step. And we never observed it in experiments with fewer workers. What the hell was going on? After literally weeks of debugging and experimenting, finally some insight. I started to log everything with the maximum verbosity possible. Interestingly, the training stop always coincided with a debug message concerning some retry. It did not always stop when such a message appeared, but whenever it stopped such a message was there. We used a feature called streaming in the the Hugging Face datasets library to read the data on the fly. It has a retry mechanism when reading the next chunk from a file fails. It turns out that when many workers are reading from the same file and one worker fails and tries to read again, there is a tiny chance for a deadlock. The more workers there are, the higher the chance of a deadlock, which also explains why we never observed this in smaller experiments. The retries could easily be avoided by changing some streaming settings, and later we switched to having a single data processing worker for iterable datasets later, which resolved the issue altogether and also improved training efficiency by up to 25%! My main takeaway from this: I totally underappreciated how hard and stressful the expensive debugging sessions are at scale. Every experiment is expensive and might take a long time to fail which limits the iteration cycle considerably. Distributed systems also behave very differently to simple single process programs, and it takes some time to adapt the mental model. If you are curious about the what it takes to train such a model, checkout the blog post for a brief tutorial on training CodeParrot : https://huggingface.co/blog/codeparrot Also, all code is open source, free to use, and available here: https://github.com/huggingface/transformers/tree/master/exam... If you are interested in more details about the design considerations when setting up a large dataset, building efficient tokenizers, and architecture choices, make sure you have a look at the CodeParrot chapter in the upcoming book on Transformers and NLP: https://learning.oreilly.com/library/view/natural-language-p...
In CodeParrot: Train and evaluate your own CoPilot model
VictorSh·Oct 18, 2021
(author here) I don't have exact numbers for latency but the inference widget is currently on a TPU v3-8 (which if I am not mistaken could roughly be compared to a cluster of 8 V100). That gives you a rough idea of the latency for short inputs. Note that a colleague just reminded me that it is possible on a single (big) GPU with enough CPU to run inference for T5-11B (which is the size we use) with offloading -> https://github.com/huggingface/transformers/issues/9996#issu...
In T0* – Series of encoder-decoder models trained on a large set of different tasks
avatar042·Sep 15, 2021
Thanks for the info on markets. What made you consider fine-tuning further on your own data? Was CLIP not sufficiently good enough to test the market? FWIW I recall having seen something similar with Google Cloud's Video Intelligence API (https://towardsdatascience.com/building-an-ai-powered-search...). Building something generic would make it especially hard to get right, especially if your users want high precision-recall from their search results. Re: licensing, the world of startups is somewhat of a wild-west these days with folks offering pre-trained models as-a-service without really thinking about the licensing implications (both on the dataset and model front). Huggingface is a classic example, and they seem to suggest that it's perfectly OK to fine-tune and use commercially (https://github.com/huggingface/transformers/issues/3357
Read the rest
#issu...), but I'm not certain that their lawyers would put it the same way.
In Show HN: Tingy – search videos with natural language queries
minimaxir·Mar 29, 2021
As someone who works on a Python library solely devoted to making AI text generation more accessible to the normal person (https://github.com/minimaxir/aitextgen ) I think the headline is misleading. Although the article focuses on the release of GPT-Neo, even GPT-2 released in 2019 was good at generating text, it just spat out a lot of garbage requiring curation, which GPT-3/GPT-Neo still requires albeit with a better signal-to-noise ratio. Most GPT-3 demos on social media are survivorship bias. (in fact OpenAI's rules for the GPT-3 API strongly encourage curating such output) GPT-Neo, meanwhile, is such a big model that it requires a bit of data engineering work to get operating and generating text (see the README: https://github.com/EleutherAI/gpt-neo ), and it's unclear currently if it's as good as GPT-3, even when comparing models ap
Read the rest
ples-to-apples (i.e. the 2.7B GPT-Neo with the "ada" GPT-3 via OpenAI's API). That said, Hugging Face is adding support for GPT-Neo to Transformers (https://github.com/huggingface/transformers/pull/10848 ) which will help make playing with the model easier, and I'll add support to aitextgen if it pans out.
In The makers of Eleuther hope it will be an open source alternative to GPT-3
moyix·Jan 24, 2021
For large models it does help! The training loop for multiple GPUs with data parallelism is roughly: 1. Split the data up 2. Do a forward and backward pass on each GPU individually 3. Compute the average of the gradients and update the model on each GPU 4. Repeat For step 3 you need to send the gradients from each GPU somewhere, and then send back either the averaged gradient or the updated model weights. So when the model is large (say, 3GB for GPT 774M!) that's a lot of GPU-GPU communication! You're right that for the vast majority of ML cases, the models are small enough that the synchronization cost is negligible, though. I wrote up some benchmarks here: https://github.com/huggingface/transformers/issues/9371
In Replicating GPT-2 at Home
moyix·Jan 23, 2021
Linux (Ubuntu 20.04) + Cuda 11.2. For the backend I use PyTorch; Tensorflow has some nice optimizations (like XLA, which uses LLVM to JIT optimized code for the GPU), but I found it very painful to get working reliably, and most of the language modeling stuff I've seen uses PyTorch. For the language model training itself I've been experimenting with a few different things. I started off with Huggingface because it's very easy to get up and running, and I still use its tokenizers library to do BPE training on the C source dataset (though there are still some hitches there – other libraries expect slightly different formats for the tokenizer model, like using different ways to represent the <|endoftext|> marker). After prototyping the C language model training at home, I tried moving the training up to NYU's HPC cluster, which has a bunch o
Read the rest
f 4xV100 and 4xRTX8000 nodes (mainly because the sound of two powerful GPU fans running at 100% gets a bit old after a while). Unfortunately I discovered that with larger models the GPU-GPU communication overhead can be prohibitive (most of the cluster nodes only support P2P GPU communication over PCIe, which is a lot slower than NVLink), and Huggingface's implementation actually performed worse on multiple GPUs than on two 3090s with NVLink (I opened an issue track it here https://github.com/huggingface/transformers/issues/9371 ). Currently I'm working on getting DeepSpeed running so that I can hopefully get better scaling even in the absence of a fast GPU-GPU interconnect. This is again a little bit annoying, because it seems like every framework wants a slightly different way of representing the tokenizer and training data – I've had to preprocess the dataset in about 4 different ways (plain text, loose JSON, npy (for DeepSpeed), and a custom indexed binary format for Megatron-LM). I'm also hoping to try out Huggingface's recently-released DeepSpeed integration, which (if it works) would be a really nice combination of usability and performance: https://huggingface.co/blog/zero-deepspeed-fairscale As for other software stack hitches: so, so many. The main one is just managing the different versions of CUDA. The 3090 is only supported starting with CUDA 11.1, but many packages and frameworks only support 11.0 at best. And some of the newer things like DeepSpeed use PyTorch extensions, which require you to have the exact version of CUDA around that was used to build PyTorch. So I've had to do a fair bit of compiling packages from source rather than relying on prebuilt packages. The path of least resistance here is probably to use the NVIDIA NGC containers, but it took NVIDIA more than a month to get them updated after the 3090 was released, and I find working inside containers for everything inconvenient anyway (I hate losing my bash history, and I always accidentally end up losing data or local changes when I exit a container). Anyway, this ended up being a bit more rambling than I intended, but it was helpful to write it all down and maybe it'll help someone else avoid some stumbling blocks :)
In Replicating GPT-2 at Home

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.