autopen

I'm working on a text editor (https://github.com/blackhole89/autopen/) that continuously analyses the buffer with a local LLM to compute token surprisal and generate candidate completions starting from any point, and switch back and forth between different ones by walking a tree structure. This is pretty different from the usual way people interact with LLMs, and has lots of interesting applications - for example, if you are using them to translate and don't like a particular word choice, you can "dig through" top alternatives on the spot or even insert your own. Applying the same approach to chain-of-thought reasoning gave me the feeling that I might be looking at a form of realistic UX for some sort of science-fiction neural AI augmentation - you can let the CoT run on and do its thing, but also interject at any point and insert a "thou

This is a program I have been working on that lets you interact with an LLM (in llama.cpp's GGUF format) locally using a text editor buffer. It tokenises the buffer on the fly, visualises the model's conditional probabilities for every token, and lets you view and jump back and forth between generated or manually entered continuations in descending order of probability at every point. Compared to some other tools that were posted here recently, it operates at a rather lower level - the branching points of the tree are single tokens=words/word fragments rather than "prompts" and "responses", and you can see the probability/surprisal levels for every single token of a text that you can edit, which opens up some interesting applications. I have made a video showcasing its use to investigate and debug a small DeepSeek-R1 distill's reasoning

I wonder how the "ring context" works under the hood. I have previously had (and recently messed around with again) a somewhat similar project designed for a more toy/exploratory setting (https://github.com/blackhole89/autopen - demo video at https://www.youtube.com/watch?v=1O1T2q2t7i4), and one of the main problems to address definitively is the question of how to manage your KV cache cleverly so you don't have to constantly perform too much expensive recomputation whenever the buffer undergoes local changes. The solution I came up with involved maintaining a tree of tokens branching whenever an alternative next token was explored, with full LLM state snapshots at fixed depth intervals so that the buffer would only have to be "replayed" for a few tokens when something changed. I wonder if there are some mathematical properties of how the

This is a simple text editor, made using gtkmm 3 and llama.cpp, that allows you to explore the possible continuations (ranked by descending probability) that an LLM would output after each token. I was quite surprised that there didn't seem to be a tool like that out there yet, so I decided to make my own. Source is on Github (https://github.com/blackhole89/autopen), though the code is still in a very rough shape.