llama-models

A lot of model-specific tokenizers have reference implementations ([0], [1]). Underlying them is a core algorithm like SentencePiece or Byte-pair encoding (BPE). Tiktoken and TokenDagger are BPE implementations. The wrapping "tokenizer" mostly deals with the quirks of the vocabulary and handling special tokens. For this project, I think there is value in building some of these model-specific quirks into the library. Could see some minor performance gains and generally make it easier to integrate with. It's probably not too much work to keep up with newer models. Tokenizers change much less frequently. [0] https://github.com/meta-llama/llama-models/blob/01dc8ce46fec... [1] https://github.com/mistralai/mistral-common/tree/main/src/mi...

The article talks about the fact that those model are "available to anyone", even Llama is Open Source from that point of view: https://github.com/meta-llama/llama-models Where Llama is not Open Source is in total respect of certain ethical principles: https://opensource.org/blog/metas-llama-license-is-still-not... So, it’snot 100% Open Source inside the context of the article it’s acceptable. Moreover, even DeepSeek is not OpenSource, it’s "OpenWeight"

Yes? I can't use the licensed model to train content? In the same way I can't use BSL to use the software in the cloud or Meta's model license when I can't be a big company. * BSL - https://github.com/clockworklabs/SpacetimeDB/blob/master/LIC... * Meta - https://github.com/meta-llama/llama-models/blob/main/models/...

I'd pump the brakes a bit on the 10M context expectations. Its just another linear attention mechanism with rope scaling [1]. They're doing something similar to what cohere did recently, using a global attention mask and a local chunked attention mask. Notably the max sequence length in training was 256k, but the native short context is still just 8k. I'd expect the retrieval performance to be all over the place here. Looking forward to seeing some topic modeling benchmarks run against it (ill be doing so with some of my local/private datasets). [1] https://github.com/meta-llama/llama-models/blob/eececc27d275... EDIT: should be fair/complete and note they do claim perfect NIAH text retrieval performance across all 10M tokens for the Scout model on their blog post: https://ai.meta.com/blog/llama-4-multimodal-intelligence/. There are some s

Meta’s Llama models (and likely many others') have similar restrictions. Since they don’t fully comply with EU AI regulations, Meta preemptively disallows their use in those regions to avoid legal complications: “With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models” https://github.com/meta-llama/llama-models/blob/main/models/...

>> Llama 3.3 70B and Qwen 2.5 72B are certainly comparable to GPT-4 > I'm skeptical; the llama 3.1 405B model is the only comparable model I've used, and it's significantly larger than the 70B models you can run locally. Every new Llama generation achieved to beat larger models of the previous generation with smaller ones. Check Kagi's LLM benchmark: https://help.kagi.com/kagi/ai/llm-benchmark.html Check the HN thread around the 3.3 70b release: https://news.ycombinator.com/item?id=42341388 And their own benchmark results in their model card: https://github.com/meta-llama/llama-models/blob/main/models%... Groq's post about it: https://groq.com/a-new-scaling-paradigm-metas-llama-3-3-70b-... Etc

Model card: https://github.com/meta-llama/llama-models/blob/main/models/... On the Models Table: https://lifearchitect.ai/models-table/

It might be a mistake since it's different from what's stated in their readme: https://github.com/meta-llama/llama-models/blob/main/models/... (which is referring to the license of Meta Llama 3.2)

LLaMa 3.1 has been pre-trained on 15 trillion tokens, plus some more millions for the fine-tuning. About 60 terabytes. https://github.com/meta-llama/llama-models/blob/main/models/... The heaviest quantised LLaMa 3.1 8B is about 3.4GB. So 0.005% compression rate, if you don't mind the intelligence of a heavily quantised 8B model.

Full text: https://github.com/meta-llama/llama-models/blob/main/models/... https://github.com/meta-llama/llama-models/blob/main/models/... > With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models.

They are not allowed indeed: https://github.com/meta-llama/llama-models/blob/main/models/... > With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models. Interesting though, since (some) EU law applies outside the EU anyway, so I'm not sure how much lawyer there is in the text.

It's not precise. People who want to use "open weights" instead of "open source" are focusing on the wrong thing. The weights are, for all practical purposes, source code in their own right. The GPL defines "source code" as "the preferred form of the work for making modifications to it". Almost no one would be capable of reproducing them even if given the source + data. At the same time, the weights are exactly what you need for the one type of modification that's within reach of most people: fine-tuning. That they didn't release the surrounding code that produced this "source" isn't that much different than a company releasing a library but not their whole software stack. I'd argue that "source" vs "weights" is a dangerous distraction from the far more insidious word in "open source" when used to refer to the Llama license: "open". The L

latest llama 3.1 is in a different repo, https://github.com/meta-llama/llama-models/blob/main/models/... , but yes, the code is shared. It astonishing that in software 2.0 era, powerful applications like llama has only hundreds of lines of code, and most work hidden in training data. Source code alone is no longer that informative as Software 1.0