transformers

>> disadvantage of Transformers codebase using the copy-paste method for models, where this fix needs to be applied to every single model separately What are the best tools we have available for tackling this kind of large scale copy-paste change? https://github.com/huggingface/transformers/pull/34191/commi... This feels too complex to tackle with PyCharm structural find and replace, even a more powerful structural find and replace like https://comby.dev/ feels underpowered here. Sourcegraph batch changes? That solves broadcasting the change but doesn’t help with capturing the change to make. Open rewrite? The python implementation is early stages, not prod ready as I understand it. Plus this change is too complex to use refaster templates even if we could use orw so you’d be debugging a fairly involved method visitor which in this case i

Are you implying that he is adjusting his reading list to fake competence? Thomas Wolf has the second highest number of commits on HuggingFace/transformers. He is clearly competent & deeply technical https://github.com/huggingface/transformers/

Sure, knowing the basics of LLM math is necessary. But it's also _enough_ to know this math to fully grasp the code. There are only 4 concepts - attention, feed-forward net, RMS-normalization and rotary embeddings - organized into a clear structure. Now compare it to the Hugginface implementation [1]. In addition to the aforementioned concepts, you need to understand the hierarchy of `PreTrainedModel`s, 3 types of attention, 3 types of rotary embeddings, HF's definition of attention mask (which is not the same as mask you read about in transformer tutorials), several types of cache class, dozens of flags to control things like output format or serialization, etc. It's not that Meta's implementation is good and HF's implementation is bad - they pursue different goals in their own optimal way. But if you just want to learn how the model wor

Fascinating work, very promising. Can you summarise how the model in your paper differs from this implementation of xLSTM ? https://github.com/huggingface/transformers/issues/27011

It seems Sepp Hochreiter has talked already about this model since Oct 2023: https://github.com/huggingface/transformers/issues/27011 In the scaling law comparison, I wonder if it is reasonable to compare number of parameters between Llama, Mamba, RWKV, xLSTM? Isn't compute time more relevant? E.g. in the figure about scaling laws, replace num of params by compute time. Specifically, the sLSTM has still recurrence (memory mixing) in it, i.e. you cannot fully parallelize the computation. So scaling up Transformer could still look better when you look at compute time. It seems neither the code nor the model params are released. I wonder if that will follow.

Is t5x an encoder/decoder architecture? Some more general options. The Flax ecosystem https://github.com/google/flax?tab=readme-ov-file or dm-haiku https://github.com/google-deepmind/dm-haiku were some of the best developed communities in the Jax AI field Perhaps the “trax” repo? https://github.com/google/trax Some HF examples https://github.com/huggingface/transformers/tree/main/exampl... Sadly it seems much of the work is proprietary these days, but one example could be Grok-1, if you customize the details. https://github.com/xai-org/grok-1/blob/main/run.py

The HuggingFace transformers library already has support for a similar method called prompt lookup decoding that uses the existing context to generate an ngram model: https://github.com/huggingface/transformers/issues/27722 I don't think it would be that hard to switch it out for a pretrained ngram model.

Not quite a layman's explanation, but if you're familiar with the implementation(s) of vanilla decoder only transformers, mixture-of-experts is just a small extension. During inference, instead of a single MLP in each transformer layer, MoEs have `n` MLPs and a single layer "gate" in each transformer layer. In the forward pass, softmax of the gate's output is used to pick the top `k` (where k is < n) MLPs to use. The relevant code snippet in the HF transformers implementation is very readable IMO, and only about 40 lines. https://github.com/huggingface/transformers/blob/main/src/tr...

I was asking on Twitter if Aaron had any experiments for transformers, since they provided some graphs for CNNs and the like, but no transformers. * Aaron et al's past work on D-Adaptation won a best ICML paper, with their follow up work being Prodigy - but both on transformers did similar or worse than AdamW. https://twitter.com/danielhanchen/status/1775547139248341125 * Superconvergence + LR range finder + Fast AI's Ranger21 optimizer was the goto optimizer for CNNs, and worked fabulously well, but on transformers, the learning rate range finder sadi 1e-3 was the best, whilst 1e-5 was better. However, the 1 cycle learning rate stuck. https://github.com/huggingface/transformers/issues/16013 * A huge issue is this needs tuning??! But how about a well tuned AdamW? Eg see https://twitter.com/kellerjordan0/status/1776716388037529843 which ou

Thanks! :) I'm pushing them into transformers, pytorch-gemma and collabing with the Gemma team to resolve all the issues :) The RoPE fix should already be in transformers 4.38.2: https://github.com/huggingface/transformers/pull/29285 My main PR for transformers which fixes most of the issues (some still left): https://github.com/huggingface/transformers/pull/29402