exllamav2

for llama 3-8b (smaller model) you need 8GB/16GB of memory: minimum/recommended || for llama 3-70b (larger model) you need 64GB/96GB of memory what kind of memory? there are 3 types of memory: GPU memory: medium capacity | high speed || Apple Unified memory: high capacity | high speed || CPU memory: high capacity | low speed || rtx 3090 has 24GB of memory at 935GB/s memory bandwidth || rtx 4060ti has 16GB of memory at 288GB/s memory bandwidth || m2 mac mini has 8GB/16GB of memory with 100GB/s memory bandwidth || m3 macbook pro has 32-128GB of memory with 400GB/s memory bandwidth || new x86 consumer cpu have up to 256GB of DDR5 at 100GB/s memory bandwidth || old cpu with ddr4 and ddr3 are much slower || || Remember, there is a simple formula for tokens/s: you need to divide memory bandwidth by model size, so for example: mac mini runs lla

There are already some implementations out there which attempt to accomplish this! Here's an example: https://github.com/silphendio/sliced_llama A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587... Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275 And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...

Without batching, I was actually thinking that's kind of modest. ExllamaV2 will get 48 tokens/s on a 4090, which is much slower/cheaper than an H100: https://github.com/turboderp/exllamav2#performance I didn't test codellama, but the 3090 TI figures for other sizes are in the ballpark of my generation speed on a 3090. 100 tokens/s batched throughput (for each individual user) is much harder.

How does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama https://github.com/turboderp/exllamav2

> brute aggressive quantization Quantization like ExLlama's EX2 or even llama.cpp's k-quants is already far from brute force: https://github.com/turboderp/exllamav2#exl2-quantization > The format allows for mixing quantization levels within a model to achieve any average bitrate between 2 and 8 bits per weight. Moreover, it's possible to apply multiple quantization levels to each linear layer, producing something akin to sparse quantization wherein more important weights (columns) are quantized with more bits. The same remapping trick that lets ExLlama work efficiently with act-order models allows this mixing of formats to happen with little to no impact on performance. Parameter selection is done automatically by quantizing each matrix multiple times, measuring the quantization error (with respect to the chosen calibration data) for eac

Can anyone provide any additional details on the EXL2[0]/GPTQ[1] quantisation, which seems to be the main reason for a speedup in this model? I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it. [0] https://github.com/turboderp/exllamav2#exl2-quantization [1] https://arxiv.org/abs/2210.17323

> Is it an average between 2 and 3 over all weights? Yes I think it's an average where different quantization levels are used for different layers or weights. Here are more details about the quantization scheme: https://github.com/turboderp/exllamav2#exl2-quantization