ggml

Didn't expect this to pop up here - please ignore for now. We are preparing official semantic versioning of llama.cpp and it's almost ready, but not quite. More info about the versioning process is here: https://github.com/ggml-org/ggml/discussions/1579

I found [ggml] is really easy, educational and fun to use. It is written in C and the backbone for llama.cpp. [ggml]https://github.com/ggml-org/ggml

It’s a different repo. https://github.com/ggml-org/ggml The models are implemented by Ollama https://github.com/ollama/ollama/tree/main/model/models I can say as a fact, for the gpt-oss model, we also implemented our own MXFP4 kernel. Benchmarked against the reference implementations to make sure Ollama is on par. We implemented harmony and tested it. This should significantly impact tool calling capability. Im not sure if im feeding here. We really love what we do, and I hope it shows in our product, in Ollama’s design and in our voice to our community. You don’t have to like Ollama. That’s subjective to your taste. As a maintainer, I certainly hope to have you as a user one day. If we don’t meet your needs and you want to use an alternative project, that’s totally cool too. It’s the power of having a choice.

“The llama.cpp project is the main playground for developing new features for the ggml library” --https://github.com/ggml-org/llama.cpp “Some of the development is currently happening in the llama.cpp and whisper.cpp repos” --https://github.com/ggml-org/ggml

One of the core design goals Georgi Gerganov had with GGUF was to not need other files. It's literally bullet point #1 in the specs >Single-file deployment >Full information: all information needed to load a model is contained in the model file, and no additional information needs to be provided by the user. https://github.com/ggml-org/ggml/blob/master/docs/gguf.md We literally just got rid of that multi file chaos only for ollama to add it back :/

Summary at https://github.com/ggml-org/ggml/pull/1103

I was under the impression that it was simply the file format used by llama.cpp and ggml, name inspired by the name of the author (https://github.com/ggerganov): https://github.com/ggerganov/ggml/blob/master/docs/gguf.md He prefixes everything with “gg” (his initials). EDIT: Confirmed: https://github.com/ggerganov/ggml/issues/220 The UF stands for Unified Format.

Cool. I was just learning about GGUF by creating my own parser for it based on the spec https://github.com/ggerganov/ggml/blob/master/docs/gguf.md (for educational purposes)

If you don't care about the details of how those model servers work, then something that abstracts out the whole process like LM Studio or Ollama is all you need. However, if you want to get into the weeds of how this actually works, I recommend you look up model quantization and some libraries like ggml[1] that actually do that for you. [1] https://github.com/ggerganov/ggml

Do you know how much faster llama.cpp would go on something like an Intel Core i9 (has AVX2 but not AVX512) when it's compiled using `cmake .. -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=Intel10_64lp -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx`? Are we talking like 10% faster inference, or 100%? Right now I'm reasonably certain llamafile is doing about the best job it can be doing on Intel/AMD, supporting SSSE3-only, AVX-only, and AVX2+F16C+FMA microprocessors at runtime. In fact, there's even an issue with the upstream llama.cpp project where they want to get rid of all the external BLAS dependencies. llama.cpp authors claim their quantization trick has actually enabled them to outdistance things like cuBLAS and I'd assume MKL too, which at best, can only operate on f32 and f16. https://github.com/ggerganov/ggml/issues/293 My concern with M

Apparently this guy like a bunch of others like https://github.com/ggerganov/ggml are implementing transformers from papers for people that want them. Pretty cool.

ggml.cpp with blast backend could be one example of it See for instance: https://github.com/ggerganov/ggml/blob/57c468b8655f3630d1749... which are the parts not available in blast

https://github.com/ggerganov/ggml TinyGrad is also targeting CPU inference, and IIRC it works ok in Apache TVM. One note is that prompt ingestion is extremely slow on CPU compared to GPU. So short prompts are fine (and tokens can be streamed once the prompt is ingested), but long prompts feel extremely sluggish. Another is that CPUs with more than 128-bit DDR5 memory busses are very expensive, and CPU token generation is basically RAM bandwidth bound.