VFvforno·Sep 21, 2026Submission
Brio mode in Colibri: Scoring a closed set instead of generating
1 points · Open discussionJustVugg
colibri
C2026–2026
JustVugg / colibri
colibri
A hierarchical inference engine for frontier MoE models: hot experts on the GPU (CUDA, Vulkan), warm ones in RAM, cold ones streamed from NVMe. Written in C, runs on the hardware you already own.
16Distinct people
18Mentions
0Explicit recommenders
50.3KGitHub stars
What to know before exploring
What it is
A hierarchical inference engine for frontier MoE models: hot experts on the GPU (CUDA, Vulkan), warm ones in RAM, cold ones streamed from NVMe. Written in C, runs on the hardware you already own.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
- antirez/ds4 · linked in 2 of the same discussions
- danveloper/flash-moe · linked in 2 of the same discussions
Mentioned in the same Hacker News threads. That is not proof they are substitutes.
This week on Hacker News
Nov 7–Nov 14 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/14/2026, 3:56:07 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by week18 dated mentions
Observed mentions: 2026-07-09 08:05:04 UTC to 2026-09-21 11:20:45 UTC. Weeks start Monday, UTC.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| Jul 6 | 2026-07-06 00:00:00 UTC | 2026-07-13 00:00:00 UTC | 3 | 1 | 2 |
| Jul 13 | 2026-07-13 00:00:00 UTC | 2026-07-20 00:00:00 UTC | 1 | 0 | 1 |
| Jul 20 | 2026-07-20 00:00:00 UTC | 2026-07-27 00:00:00 UTC | 2 | 0 | 2 |
| Jul 27 | 2026-07-27 00:00:00 UTC | 2026-08-03 00:00:00 UTC | 6 | 0 | 6 |
| Aug 3 | 2026-08-03 00:00:00 UTC | 2026-08-10 00:00:00 UTC | 0 | 0 | 0 |
| Aug 10 | 2026-08-10 00:00:00 UTC | 2026-08-17 00:00:00 UTC | 2 | 1 | 1 |
| Aug 17 | 2026-08-17 00:00:00 UTC | 2026-08-24 00:00:00 UTC | 1 | 0 | 1 |
| Aug 24 | 2026-08-24 00:00:00 UTC | 2026-08-31 00:00:00 UTC | 1 | 0 | 1 |
| Aug 31 | 2026-08-31 00:00:00 UTC | 2026-09-07 00:00:00 UTC | 1 | 0 | 1 |
| Sep 7 | 2026-09-07 00:00:00 UTC | 2026-09-14 00:00:00 UTC | 0 | 0 | 0 |
| Sep 14 | 2026-09-14 00:00:00 UTC | 2026-09-21 00:00:00 UTC | 0 | 0 | 0 |
| Sep 21 | 2026-09-21 00:00:00 UTC | 2026-09-28 00:00:00 UTC | 1 | 1 | 0 |
What Hacker News says
18 sourcesColibri did that first for GLM-5.2 https://github.com/JustVugg/colibri
In Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/sSpeaking as someone who isn't really well versed in this, does 18B active parameters mean that you could potentially hold only the 18B parameters in RAM and stream the rest from a fast NVMe SSD for acceptable performance similar to how Colibri works?
https://github.com/JustVugg/colibri
In GLM-5.3-FlashThis is interesting. I wonder how it could work with something like https://github.com/JustVugg/colibri.
In Show HN: Shoehorn – Quantize any model down to run on your machineThe thing that blows me away is it does this at one quarter the total parameter count of K3 (and 40% active parameter count). There's plenty of room at the bottom.
> How are you all toying with running this kind of thing in a mega quantized way locally?
Sure, let me answer that in excessive detail. I briefly tried running the UD IQ3_S quant of GLM-5.2, which is 288 GiB of weights (301 GB). Setup was: llama.cpp, 1x NVMe SSD (Evo 980), 64 GiB DDR5-5200, i9-13900HX, and 1x RTX Pro 6000. Token generation around 0.7 t/s. Not remotely usable interactively, but something I could plausibly push a codebase into and come back to a review in a couple of days.
There's potential for that hardware to go much faster, but current local inference backends make poor use of the memory hierarchy. Ideally I would have: always-active weights, KV and hot expert
Read the rest
cache in VRAM; warm expert victim cache in host RAM; and disk as a last resort. Instead it's 1/3rd of the layers fully pinned in VRAM (all experts), and 2/3rds running wholly on the CPU with mmap()'d weights. The CPU cores spend most of their time sleeping on disk fills.
llama.cpp has backed itself into a bit of a corner architecturally by trying to support all models on all possible backends. If you look into how their "MoE offload" feature works (not viable for me because it requires enough host RAM to permanently pin the weights) you very quickly realise it's "oops, all bubbles!" due to the static compute graph splits. There are more focused frameworks like DS4 [1] and Colibri [2] which have better support for streaming weights from disk, and support GLM-5.2.
Obviously I wouldn't recommend my setup for huge models like GLM-5.2. Supposedly it can just about be squeezed into 3x GB10, or run comfortably on 4x GB10 (tensor-parallel) for multi-user serving. I'm not sure whether that qualifies as local, but it's at least not a rack.
[1] https://github.com/antirez/ds4
[2] https://github.com/JustVugg/colibri
Call for probes: building the GLM-5.2 Expert Atlas
1 points · Open discussionCheck for colibri, dwarf star and flash-moe. they do similar things with bigger models
https://github.com/JustVugg/colibri
https://github.com/antirez/ds4
https://github.com/danveloper/flash-moe
In Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series MacYeah! Check the Colibri and Flash-MoE projects. They’re already doing that.
https://github.com/danveloper/flash-moe
https://github.com/JustVugg/colibri
In Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series MacI'm really excited about what's been happening couple last weeks for local inference. I feel like it all started after colibri [1] was released.
Great work !
Anyone got recommendation about what local model to use for what purpose ?
I feel like (as they were saying in moonshot blog post [2]) each llm can be an expert in its own categories and with several small local we might get good coverage for decent usage, granted each one is specialized enough.
[1] : https://github.com/JustVugg/colibri
[2] : https://fireworks.ai/blog/kimik3-fable
In Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series MacI haven't tried it, but colibri is meant to allow running GLM 5.2 the 744B model in 32GB by using prefetches and streaming from fast SSDs.
https://github.com/JustVugg/colibri
In Chinese chipmaker shares surge 470%> running it on a no GPU, but tons of RAM server
Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable.
[1]: https://github.com/JustVugg/colibri
In Kimi-K3 on HuggingFaceGiven the frontier-level capabilities of Kimi K3, I'm wondering if it's possible to extract the core capabilities (fundamental reasoning and tool calling) of the model into a smaller one that consumer devices could run? Not sure exactly how, but either by heavy distillation or some other surgical method since Kimi has a Mixture of Experts architecture.
I think it's very valuable to have a smaller model that doesn't have any domain knowledge or facts built into its weights, but given the right context, could accurately reason about what to do and use the right tools.
I'm aware of colibri [1], but so far I've only seen extremely slow performance.
[1] https://github.com/JustVugg/colibri
In Kimi-K3 on HuggingFaceSomeone did this exact thing recently, but running GLM-5.2 with something like 16GB of DRAM, a standard desktop CPU and nVME SSD.
I think they got something like 10 _seconds per token_ (not tokens per second).
EDIT: it was 25GB of ram and up to 20 seconds per token!
https://github.com/JustVugg/colibri
In Running a 28.9M parameter LLM on an $8 microcontrollerAlready is possible. On a machine with 32GB ram, and NO gpu. Just need a large SSD or NVME. Streams from disk to memory.
https://github.com/JustVugg/colibri
In Who's afraid of Chinese models?If you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibri
It says so right in the readme. They’re not hiding anything.
In Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPUShowing the 15 most recent of 18 sources. Enable JavaScript to search and browse every indexed mention.