mamba

This model was discussed recently: https://news.ycombinator.com/item?id=38522428 It's a new kind of ML model architecture that can be used instead of a transformer in LLMs. See also the original repo from the paper: https://github.com/state-spaces/mamba

This is a reddit post which links to a Twitter post [0] which links to this paper [1] and this repo [2] [0] - https://nitter.net/_albertgu/status/1731727672286294400 [1] - https://arxiv.org/abs/2312.00752 [2] - https://github.com/state-spaces/mamba Out of curiosity, does anyone feel as though there's any benefit to linking to reddit when we can link to whatever the reddit post actually is? I for one do not click the link and read discussion on reddit - if I wanted that sort of discussion, I would browse there, not HN.

Hey everyone! Many of you might have come across the Mamba paper a few days ago, which introduced an LLM based on a state space model architecture. The Mamba architecture is quite useful as its complexity scales subquadratically with input length and is therefore way more efficient than transformer models: https://github.com/state-spaces/mamba I got really excited about the paper, so I decided to fine-tune the model on a chat dataset. It turns that this actually worked quite well! The model is quite suitable for casual chatting, which honestly surprised me given that it only has 2.8B parameters and the base model was only trained on the Pile. It's quite exciting that we might have a serious candidate for an architecture that could dethrone transformers. You can find both my fine-tuning and inference code here: https://github.com/havenhq/m

Implementation of selective_scan_cuda: https://github.com/state-spaces/mamba/blob/6dbfc4553a98c81e1...

I've been doing web related engineering for the past few years and recently been interested in machine learning. Taking a random perusal through the code for this [1] feels daunting. So many single letter variables, it looks like old JS soup code. It makes me hesitant to leave Typescript and even Rust (for web tooling). [1] https://github.com/state-spaces/mamba/blob/main/mamba_ssm/op...