HackerReposA FIELD GUIDE TO OPEN SOURCE
ProSponsor the shelf♧ 0
← Back to top all-time
dipampaul17
KVSplit
Python2025–2025
dipampaul17 / KVSplit

KVSplit

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

2Distinct people
2Mentions
0Explicit recommenders
362GitHub stars
REPOSITORY AT A GLANCE

What to know before exploring

What it is

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

GitHub

Where it is used

No documented product use has been verified yet.

What people are saying

No recommendation phrase was found in the indexed comments.

No classified warning is attached to this page. Read the discussions before treating silence as approval.

Also in these discussions

No other repository is repeatedly linked in the same discussions.

This week on Hacker News

Nov 9–Nov 16 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/16/2026, 3:59:04 AM UTC.

No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.

Mentions over time

HN mentions by hour2 dated mentions

Observed mentions: 2025-05-16 20:04:58 UTC to 2025-05-17 05:15:25 UTC.

012025-05-16 20:00:00 UTC to 2025-05-16 21:00:00 UTC (exclusive): 1 mentions; 1 submissions, 0 commentsMay 16, 20:002025-05-16 21:00:00 UTC to 2025-05-16 22:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-16 22:00:00 UTC to 2025-05-16 23:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-16 23:00:00 UTC to 2025-05-17 00:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-17 00:00:00 UTC to 2025-05-17 01:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 commentsMay 17, 00:002025-05-17 01:00:00 UTC to 2025-05-17 02:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-17 02:00:00 UTC to 2025-05-17 03:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-17 03:00:00 UTC to 2025-05-17 04:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-17 04:00:00 UTC to 2025-05-17 05:00:00 UTC (exclusive): 0 mentions; 0 submissions, 0 comments2025-05-17 05:00:00 UTC to 2025-05-17 06:00:00 UTC (exclusive): 1 mentions; 0 submissions, 1 commentsMay 17, 05:00

Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.

View mention counts and date ranges
UTC ranges include their start and exclude their end. Select a period to filter the source evidence.
PeriodStart (UTC)End (exclusive, UTC)MentionsSubmissionsComments
May 16, 20:002025-05-16 20:00:00 UTC2025-05-16 21:00:00 UTC110
May 16, 21:002025-05-16 21:00:00 UTC2025-05-16 22:00:00 UTC000
May 16, 22:002025-05-16 22:00:00 UTC2025-05-16 23:00:00 UTC000
May 16, 23:002025-05-16 23:00:00 UTC2025-05-17 00:00:00 UTC000
May 17, 00:002025-05-17 00:00:00 UTC2025-05-17 01:00:00 UTC000
May 17, 01:002025-05-17 01:00:00 UTC2025-05-17 02:00:00 UTC000
May 17, 02:002025-05-17 02:00:00 UTC2025-05-17 03:00:00 UTC000
May 17, 03:002025-05-17 03:00:00 UTC2025-05-17 04:00:00 UTC000
May 17, 04:002025-05-17 04:00:00 UTC2025-05-17 05:00:00 UTC000
May 17, 05:002025-05-17 05:00:00 UTC2025-05-17 06:00:00 UTC101

What Hacker News says

1 sources

Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon

I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer context on the same Mac. Memory usage scales with sequence length, so savings compound as context grows. Implementation was straightforward: 1. Added --kvq-key and --kvq-val flags to llama.cpp 2. Applied existing quantization logic separately to K and V ten
Read the rest
sors 3. Validated with perplexity metrics across context lengths 4. Used Metal for acceleration (with -mlong-calls flag to avoid vectorization issues) Benchmarked on an M4 MacBook Pro running TinyLlama with 8K context windows. Compatible with Metal/MPS and optimized for Apple Silicon. GitHub: https://github.com/dipampaul17/KVSplit
272 points · Open discussion

MAKE YOURSELF AT HOME

A few shortcuts

/
Search the collection
?
Open this guide
Esc
Close dialogs and navigation

Your saved repositories and interests stay in this browser.