BitNet

This article (comparing the x86 AVX-512F instruction VPTERNLOGD to Amiga's Blitter) is not the end of the discussion for this instruction, it's actually the beginning... See, VPTERNLOGD is a ternary bitwise instruction. In theory, that means that it could (in theory, temporarily ignoring other factors like streaming, parallelism, memory access speed, etc.) replace the Matrix Multiply (aka "MatMul") component of LLM's, if, if and only if the weights in an LLM are changed to ternary (-1,0,1)... Matrix Multiply is at the core of all GPU's and AI accelerators -- so alternatives to it, such as VPTERNLOGD (especially on older/cheaper hardware, as x86 AVX-512F apparently dates back to 2016), as well as Ternary LLM's -- may be worth a little investigation... Yes, ternary LLM's do come with their own set of problems(!) -- but as a strictly-for-int

No, because the LLMs will keep getting more efficient and capable. Distillation and quantization will mean firms spending trillions on giant data centres are left holding the bag. I suspect Apple ends up laughing all the way to the bank. https://github.com/microsoft/BitNet

> You are greatly underestimating the current hardware requirements for productive local LLMs. Fixed that for you. Right now most models produced are based on floating point maths and probabilities, which is "expensive" to do math on. Microsoft has researched 1-bit LLMs which can run much more efficiently, and on much cheaper hardware[1]. If this research is reproducable and reusable outside their research models, this means the cost of running self-hosted LLMs will be reduced by an order of magnitude once this hits mainstream. [1] https://github.com/microsoft/BitNet

I don't think there is a new model, they recently released a new inference engine for CPUs for bitnet models: https://github.com/microsoft/BitNet/blob/main/src/README.md

I think the README [1] for the new CPU feature is of more interest, showing linear speedups with number of threads. Up to 73 tokens/sec with 8 threads (64 toks/s for their recommended Q6 quant): https://github.com/microsoft/BitNet/blob/main/src/README.md

It comes from (intentionally?) misleading docs: https://github.com/microsoft/BitNet/issues/391 (only suggesting that it's intentional because it's been there so long)

Paper: https://arxiv.org/pdf/2310.11453

I've been optimizing ternary operations for BitNet 1.58b and found significant overhead in the current implementation. I wrote a dependency-free C kernel (sparse-ternary-fma) using 2-bit encoding and AVX-512 instructions. Benchmarks on Intel Xeon (N=4096): Throughput (Dense): 2.38x faster (8.21 GFLOPS vs 3.45 AVX2) Throughput (Sparse 80% zeros): 26.12x faster (23.25 GFLOPS vs 0.89 Scalar) Memory: 4x denser (2-bit vs 8-bit standard) This approach packs 4 trits per byte and leverages sparsity-aware FMA to skip zero-valued weights, which is critical for 1.58-bit quantization efficiency. PR is pending on the Microsoft BitNet repo. Code is open source here:https://github.com/microsoft/BitNet/pull/365

Have you tried anything with https://codeberg.org/ikawrakow/illama https://github.com/ikawrakow/ik_llama.cpp and their 4Bit-quants? Or maybe even Microsofts Bitnet? https://github.com/microsoft/BitNet https://github.com/ikawrakow/ik_llama.cpp/pull/337 https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf ? That would be an interesting comparison for running local LLMs on such low-end/edge-devices. Or common office machines with only iGPU.

CPU inference models are also on their way which will help immensely with running LLMs locally without having to spend sheer $$ on high end GPU's. https://github.com/microsoft/BitNet?tab=readme-ov-file

Repo with demo video and benchmark: https://github.com/microsoft/BitNet "...It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption..." https://arxiv.org/abs/2402.17764

and the repo for this project: https://github.com/microsoft/BitNet