Proteus-Attention

On a raw, untrained model, I found that 334 out of 500 tokens were already being caught by this system. It's a natural, emergent behavior. 2. The Alpha Slider - "Why can't I just change my model?" I hated that I couldn't just switch my model from dense, to sparse, to linear whenever I wanted. So, I built a custom Triton kernel to do exactly that. The result is a single knob called alpha: Dense, high-fidelity? alpha = 0.0. Balanced sub-quadratic? alpha = 0.3. Screaming-fast linear time? alpha = 1.0 and the attention mechanic goes brrrrrr. 3. Chunking & RoPE - "So I got rid of it." My new systems got me far, but the VRAM bottleneck was still a headache. So I got rid of it. The idea is simple: chunking. Break a massive context into small pieces, shunt them to system RAM, and use a tiny VRAM buffer for only the most important tokens. DNA tells us what's important. As a Hail Mary, I added RoPE to preserve where it came from. This combination creates contextual teleportation. It allows the model to create a perfect "highlight reel" and reason over it as if critical facts, separated by thousands of pages, were sitting right next to each other. It's your own little wormhole across data space. TL;DR: I built an extreme context system that costs less than Minecraft to run. Would love feedback, as I'm still exploring how far it can go. Github: https://github.com/Zen-Sherbert/Proteus-Attention/tree/main