TOtosh·Dec 13, 2023Submission
HuggingFace Transformers Release v4.36: Mixtral, Llava/BakLlava, SeamlessM4T v2
5 points · Open discussionhuggingface
transformers
Python2018–2026
huggingface / transformers
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
79Distinct people
93Mentions
1Explicit recommenders
168.9KGitHub stars
What to know before exploring
What it is
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
For deep learning / neural networks, I think these are the best balance of theory and practice (not to mention great visualizations): http://cs231n.github.io More on the…
jointpdf used a recommendation phrase. One indexed comment matches. This is a phrase match, not a verdict.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- ggml-org/llama.cpp · linked in 3 of the same discussions
- minimaxir/aitextgen · linked in 2 of the same discussions
- minimaxir/gpt-2-simple · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 11:34:28 AM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year93 dated mentions
Observed mentions: 2018-11-07 08:59:39 UTC to 2026-08-06 23:03:10 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2018 | 2018-01-01 00:00:00 UTC | 2019-01-01 00:00:00 UTC | 1 | 1 | 0 |
| 2019 | 2019-01-01 00:00:00 UTC | 2020-01-01 00:00:00 UTC | 9 | 3 | 6 |
| 2020 | 2020-01-01 00:00:00 UTC | 2021-01-01 00:00:00 UTC | 15 | 3 | 12 |
| 2021 | 2021-01-01 00:00:00 UTC | 2022-01-01 00:00:00 UTC | 7 | 1 | 6 |
| 2022 | 2022-01-01 00:00:00 UTC | 2023-01-01 00:00:00 UTC | 8 | 1 | 7 |
| 2023 | 2023-01-01 00:00:00 UTC | 2024-01-01 00:00:00 UTC | 20 | 4 | 16 |
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 12 | 2 | 10 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 12 | 3 | 9 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 9 | 2 | 7 |
What Hacker News says
20 sourcesHuggingFace: Support for the Mixtral Moe
4 points · Open discussionIf you want to tinker with the architecture Hugging Face has a FOSS implementation in transformers: https://github.com/huggingface/transformers/blob/main/src/tr...
If you want to reproduce the training pipeline, you couldn't do that even if you wanted to because you don't have access to thousands of A100s.
In Mistral AI Valued at $2BToo much money being thrown around on BS in the LLM space, hardly any of it is going to places where it matters. Ignorance on the part of investors.
For example, the researchers working hard on better text sampling techniques (i.e. https://arxiv.org/abs/2202.00666), or on better constraint techniques (i.e. like this https://arxiv.org/abs/2306.03081), or on actual negative prompting/CFG in LLMs (i.e. like this https://github.com/huggingface/transformers/issues/24536) are doing far FAR more to advance the state of AI than dozens of VC backed LLM companies operating today. They are all laboring in relative obscurity.
HN, and the NLP community have some serious blindspots with knowing how to exploit their own technology. At least someone at Andreessen Horowitz got a clue and gave some funding to Oogabooga - still waiting for Automatic1111 to
Read the rest
get any funding.
I was a little confused about this too. The authors say in the paper:
"The outputs of the ViT image encoder before pooling form the visual tokens, which are linearly projected and prepended to the embedded input text tokens."
I took a look at the HuggingFace implementation of ViT [1]. After the ViT encoder blocks there's a layer norm and then a pooling layer (line 595), where the pooling layer involves taking the first token output from the layer norm and running it through a dense layer. So, it looks like in PaLI-3 the tokens are the hidden states output by the layer norm after the ViT encoder blocks.
[1] https://github.com/huggingface/transformers/blob/main/src/tr...
In PaLI-3 Vision Language ModelsHere's the model code
https://github.com/huggingface/transformers/blob/main/src/tr...
In Mistral 7BThere's no such thing as "base models have only the temperature setting". Models do not have sampling settings (temperature, repetition penalty, etc), the sampling code does, which obviously you can use on any model.
For example, here's a function from llama.cpp that applies repetition penalty: https://github.com/ggerganov/llama.cpp/blob/master/llama.cpp...
Here's the one from transformers:
https://github.com/huggingface/transformers/blob/0a55d9f7376...
To summarize how they work: you keep some number of previously generated tokens, and once you get logits that you want to sample a new token from, you find the logits for existing tokens and multiply them by a penalty, thus lowering the probability of the corresponding tokens.
In Apple’s new Transformer-powered predictive text modelVery cool. This came up in a huggingface transformers issue a while ago and we also determined memorization to be the likely reason. It's nice to see someone else reach the same conclusion.
https://github.com/huggingface/transformers/issues/18730
In Can LLMs learn from a single example?Ask HN: Are you training and running custom LLMs and how are you doing it?
I have been researching methods and projects built around training and running LLMs locally. I'm interested in what others have been using on this front (including straight up Pytorch/Transformers). Here's what I've gathered so far:
Engines/APIs:
- vllm: Inference and serving engine for LLMs (none quantized models only?) [1]
- ollama: Go project to run, create and share LLMs [2]
- llama.cpp: Inference of LLaMA models in C/C++ w/UI (including quantized models) [3]
- llama-cpp-python: run OpenAI compatible API bindings for llama.cpp [4]
- llm-engine: engine for fine-tuning and serving LLMs [5]
- Lamini: hosted? closed source? solution for training LLMs [6]
- GPT4All: free, locally run chatbot [7]
- SkyPilot: framework for running LLMs, AI, and batch jobs [8]
- HuggingFace Transformers: APIs and tools to download and train
Read the rest
models (via Pytorch) [9]
- RAGStack: Deploy a private ChatGPT alternative hosted within your VPC [14]
UI/Interface:
- Simon's `llm` tool [15]
Quantization Bits:
- AutoGPTQ: GPTQ algorithm based quantization package [10]
- QLoRA: finetuning of quantized LLMs [11]
- bitsandbytes: CUDA functions for PyTorch [12]
- SkyPilot QLoRA [13]
Video Guides:
- https://www.youtube.com/watch?v=eeM6V5aPjhk
- https://www.youtube.com/watch?v=TYgtG2Th6fI (jawerty's example)
Reference:
- [1] https://github.com/vllm-project/vllm
- [2] https://github.com/jmorganca/ollama
- [3] https://github.com/ggerganov/llama.cpp
- [4] https://github.com/abetlen/llama-cpp-python
- [5] https://github.com/scaleapi/llm-engine
- [6] https://www.lamini.ai/
- [7] https://github.com/nomic-ai/gpt4all
- [8] https://github.com/skypilot-org/skypilot
- [9] https://github.com/huggingface/transformers/releases
- [10] https://github.com/PanQiWei/AutoGPTQ
- [11] https://github.com/artidoro/qlora
- [12] https://github.com/TimDettmers/bitsandbytes
- [13] https://github.com/artidoro/qlora/pull/132
- [14] https://github.com/psychic-api/rag-stack
- [15] https://simonwillison.net/2023/Aug/1/llama-2-mac/
Introducing LLongMA, a series of OpenLLaMA models, trained at 8k context length using linear positional interpolation scaling. The model was trained in collaboration with theemozilla of NousResearch and Kaiokendev.
We worked directly with Kaiokendev, to extend the context length of the open-llama 7b and 3b models through fine-tuning. The fine-tuned models maintain the same perplexity at 8k extrapolation surpassing the performance of other recent methodologies.
Applying the method to the rotary position embedding requires only slight changes to the model's code by dividing the positional index, t, by a scaling factor.
You can review the benchmarks of the LLongMA models trained at 8k when compared to the original Open-LLaMA models trained at 2k. We show slight performance boosts on multiple benchmarks and minimal degradation on others.
The
Read the rest
LLongMA 7b model is available on huggingface to use:https://huggingface.co/conceptofmind/LLongMA-7b
The LLongMA 3b model can also be found on huggingface here:https://huggingface.co/conceptofmind/LLongMA-3b
The repository containing theemozilla’s implementation of scaled rotary embeddings can be found here:https://github.com/jquesnelle/scaled-rope
A LLongMA-13b model trained at 8k context length will be released soon. As well as a suite of LLongMA models trained at 16k and 32k context lengths.
If you would like to learn more about scaling rotary embeddings, I would strongly recommend reading Kaiokendev's blog posts on his findings:https://kaiokendev.github.io/
A PR to add scaled rotary embeddings to huggingface transformers is currently in progress: https://github.com/huggingface/transformers/pull/24653
The model was trained for ~1 billion tokens on togethercompute's Red Pajama dataset. The context length of the examples varies: https://huggingface.co/datasets/togethercomputer/RedPajama-D...
The pre-tokenized dataset will be available here for you to use soon: https://huggingface.co/datasets/conceptofmind/rp-packed-8k-n...
I would also recommend checking out the phenomenal research by OfirPress on ALiBi which laid the foundation for many of these scaling techniques: https://arxiv.org/abs/2108.12409
OfirPress just live-streamed his Ph.D. defense recently here:https://twitter.com/OfirPress/status/1677025700560388098?s=2...
It is also worth reviewing the paper, A Length-Extrapolatable Transformer, and xPos technique which also applies scaling to rotary embeddings: https://arxiv.org/pdf/2212.10554.pdf
We previously trained the first publicly available model with rotary embedding scaling here: https://twitter.com/EnricoShippole/status/165559930145459404...
The compute for this model release is all thanks to the generous sponsorship by carperai, EMostaque, and StabilityAI.
You can find out more about the NousResearch organization here:https://huggingface.co/NousResearch
The base OpenLLaMA models used in these experiments were developed by younggeng and haoliuhl and can be found here: https://huggingface.co/openlm-research
This is not an official StabilityAI or NousResearch product.
Credits to Zanzibased for the llama image.
If you have any questions about the data or models be sure to reach out and ask! I will try to respond promptly.
The author of the paper has already shared their implementation and are willing to contribute it to hf-transformers: https://github.com/huggingface/transformers/issues/24536
In Classifier Free Guidance works on LLMs with a significant boost in performance> What's a fair benchmark?
- /? pytorch tensorflow benchmarks webgpu 2023 site:github.com
https://www.google.com/search?q=pytorch+tensorflow+benchmark...
- [tfjs benchmarks]
- huggingface/transformers:src/transformers/benchmark https://github.com/huggingface/transformers/tree/main/src/tr...
In PyTorch for WebGPUWhy is GitHub search so terrible?
It's worse than useless, since it's actively misleading.
A basic, straightforward, exact match search finds nothing: https://github.com/huggingface/transformers/search?q=TFGPT2LMHeadModel
And yet, here it is: https://github.com/huggingface/transformers/blob/4d0ea3d2692ccf2e7f091031fedb1ed8a…
I saw another thread where they tried to justify not indexing forks, but in this case, this is not a fork; it's one of the most-starred repositories on the entire site.
3 points · Open discussionI'm worried about the version hell of relying on HuggingFace's transformers.
In the model card : config.json [1] and generation_config.json there are strings "transformers_version": "4.28.1" and it refers to some common architecture "GPTNeoXForCausalLM" .
Which if I follow the string soft link correctly means, it refers to this file [2] with a long history of commits including some recents [3].
I don't understand how to get out of the version hell. I can install and pin a version of the transformer library, but sooner or later if I chain different foundation models I'll have to do this for each different model card that has its own version.
And then those transformer library version are probably dependent on some pytorch versions.
I don't understand how you can't reliably track a bug, or a change in behavior, or guarantee that the model t
Read the rest
hat you spend millions of dollar training doesn't rust, is reproducible or become unusable due to this mess.
(And also the weights are de-serialized pickled python dictionary which is from a safety point of view equivalent to running untrusted binaries and very worrying on its own.)
[1]https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b/b...
[2]https://github.com/huggingface/transformers/blob/v4.28.1/src...
[3]https://github.com/huggingface/transformers/commits/v4.28.1/...
I'm a heavy user of type hints and enable pyright and mypy's strict modes whenever possible. However, you can't always be strict: if you use almost any package in the data science/ML ecosystem, you're unlikely to get good type inference and checking[1]. In those cases, it can still be useful to type some parameters and return values to benefit from _some_ checking, even if you don't have 100% coverage.
Type hints also bring improved completion, which is nice too.
[1] For example, huggingface's transformers library decided to drop support for full type checking because it was unsustainable but decided to keep the types for documentation[2]. There are stubs for pandas, but they're not enough because pandas has a tendency to change return types based on the input, and that breaks quickly.
[2] https://github.com/huggingface/transformers/pull/
Read the rest
Showing the 15 most recent of 20 sources. Enable JavaScript to search and browse every indexed mention.