exllamav2

for llama 3-8b (smaller model) you need 8GB/16GB of memory: minimum/recommended || for llama 3-70b (larger model) you need 64GB/96GB of memory what kind of memory? there are 3 types of memory: GPU memory: medium capacity | high speed || Apple Unified memory: high capacity | high speed || CPU memory: high capacity | low speed || rtx 3090 has 24GB of memory at 935GB/s memory bandwidth || rtx 4060ti has 16GB of memory at 288GB/s memory bandwidth || m2 mac mini has 8GB/16GB of memory with 100GB/s memory bandwidth || m3 macbook pro has 32-128GB of memory with 400GB/s memory bandwidth || new x86 consumer cpu have up to 256GB of DDR5 at 100GB/s memory bandwidth || old cpu with ddr4 and ddr3 are much slower || || Remember, there is a simple formula for tokens/s: you need to divide memory bandwidth by model size, so for example: mac mini runs lla