flash-moe

Check for colibri, dwarf star and flash-moe. they do similar things with bigger models https://github.com/JustVugg/colibri https://github.com/antirez/ds4 https://github.com/danveloper/flash-moe

Yeah! Check the Colibri and Flash-MoE projects. They’re already doing that. https://github.com/danveloper/flash-moe https://github.com/JustVugg/colibri

I think I first saw Flash-MoE (https://github.com/danveloper/flash-moe) in April. Huge respect to them, it was a big inspiration for this project!

Another recent project that runs a huge model on a 48gb Mac is https://github.com/danveloper/flash-moe - it gets over 5 tokens/sec on an M3 Max compared to this projects very impressive 1 token/sec on an M5 Max. So for anyone wanting to tackle a Mac only version that targets lower spec machines this looks like a good candidate with plenty of room for speedups [edit: because it doesn't use the gpu]. Not hijacking anything as this project is amazing.

Yeah, local is clearly the future. Even beyond the cheap Chinese models you can install the apfel[1] stuff if you're on a mac and want a quick available onboard cli option. And I'm sure people will adapt the Flash-MoE[2] integration to be even better soon as well. [1] https://apfel.franzai.com/ [2] https://github.com/danveloper/flash-moe

Sure! Note: these were just two that I starred when I saw them posted here. I have not looked seriously at it at the moment, https://github.com/danveloper/flash-moe https://github.com/t8/hypura

Yes, this is a reference project, the main different is we don't use os swap ( it introduces latency, will add https://github.com/danveloper/flash-moe to the original reference as well ).