transformers

You have to share your contact information, including DoB, and then be approved access, to obtain the models, and given that it's Meta I assume they're actually validating it against their All Humans database. They made their own DINOv3 license for this release (whereas DINOv2 used the Apache 2.0 license). Neat though. Will still check it out. As a first comment, I had to install the latest transformer==4.56.0dev (e.g. pip install git+https://github.com/huggingface/transformers) for it to work properly. 4.55.2 and earlier was failing with a missing image type in the config.

The new transformers release describes the model: https://github.com/huggingface/transformers/releases/tag/v4.... > GPT OSS is a hugely anticipated open-weights release by OpenAI, designed for powerful reasoning, agentic tasks, and versatile developer use cases. It comprises two models: a big one with 117B parameters (gpt-oss-120b), and a smaller one with 21B parameters (gpt-oss-20b). Both are mixture-of-experts (MoEs) and use a 4-bit quantization scheme (MXFP4), enabling fast inference (thanks to fewer active parameters, see details below) while keeping resource usage low. The large model fits on a single H100 GPU, while the small one runs within 16GB of memory and is perfect for consumer hardware and on-device applications.

Reference implementations are unmaintained and buggy. For example https://github.com/huggingface/transformers/issues/27961 OpenAI's tokenizer for CLIP is buggy, it's a reference implementation, it isn't the one they used for training, and the problems with it go unsolved and get copied endlessly by other projects. What about Flux? They don't say it was used for training, it wasn't, there are bugs with it that break cudagraphs or similar that aren't that impactful. On the other hand, it uses CLIP reference, and CLIP reference is buggy, so this is buggy...

They are still deterministic. You can set temperature to zero to get the output to be consistent, but even the temperature usually uses a seed or psuedo random number generator. Though this would depend on the implementation. https://github.com/huggingface/transformers/blob/d538293f62f...

There were actually multiple bugs which impacted long context benchmarks and general inference - I helped fix some of them. 1. RMS norm eps was 1e-6, but should be 1e-5 - see https://github.com/huggingface/transformers/pull/37418 2. Llama 4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889 3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311 If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for Scout actually did better than inference providers by +~5% on MMLU Pro. https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-... has more details

Not a visual model, but extremely excited to see how Qwen3 performs. https://news.ycombinator.com/item?id=43455084 https://github.com/huggingface/transformers/pull/36878 Qwen2.5-VL looks like an incredibly fantastically useful system, that I don't see any other publicly available models offering anything like. It makes me wonder if we really have tools capable of using it well, since imagery and video is so relatively uncommon. I wonder how we can better prompt & load multimedia data into context for these LLM.

Qwen 3 is coming imminently as well https://github.com/huggingface/transformers/pull/36878 and it feels like Llama 4 should be coming in the next month or so. That said none of the recent string of releases has done much yet to "smash a wall", they've just met the larger proprietary models where they already were. I'm hoping R2 or the like really changes that by showing ChatGPT 3->3.5 or 3.5->4 level generational jumps are still possible beyond the current state of the art, not just beyond current models of a given size.

why not just decrease </think> probability? Huggingface's transformers library supports something similar to this. You set a minimum length, and until that length is reached, the end of sequence token has no chance of being output. https://github.com/huggingface/transformers/blob/51ed61e2f05... S1 does something similar to put a lower limit on its reasoning output. End of thinking is represented with the <|im_start|> token, followed by the word 'answer'. IIRC the code dynamically adds/removes <|im_start|> to the list of suppressed tokens. Both of these approaches set the probability to zero, not something small like you were suggesting.

Transformers like Llama use rotary embeddings which are applied in every single attention layer https://github.com/huggingface/transformers/blob/222505c7e4d...