mini-AGI

B 1.23B 36 TB (9T tok) ~$900k 0.68 Chinchilla 1B 1.00B 5.6 TB ~$68k 0.90 GPT-2 XL 1.56B 40 GB ~$43k 1.07 GPT-2 Large 0.77B 40 GB ~$21k 1.12 mini-AGI (online) 1.26B 2.8 GB ~$56 1.14 GPT-2 Medium 0.35B 40 GB ~$10k 1.17 mini-AGI (dynamic) 1.26B 2.8 GB ~$56 1.23 mini-AGI (frozen) 1.26B 2.8 GB ~$56 1.29 GPT-2 Small 0.12B 40 GB ~$3k 1.33 2. PG19 (100 Project Gutenberg books, 41 MB test split): Model Params Trained On Cost PG19 (bits/byte) ↓ MambaByte-972M 0.97B 75 GB (PG19) ~$4k 0.883 MegaByte-1B 1.02B 80 GB (PG19) ~$1.6k 1.000 Transformer-320M 0.32B 80 GB (PG19) ~$1.6k 1.057 mini-AGI (online) 1.26B 2.8 GB (0 PG19) ~$56 1.447 mini-AGI (frozen) 1.26B 2.8 GB (0 PG19) ~$56 1.995 The published models were trained on PG19's own training books, 75 to 80 GB of them, and are scored on more books like those. mini-AGI has never been trained on PG19 and has read no Victorian novels, so for it these books are completely out of its domain. There are more details in the repository. The most important part is a difference between "frozen" and "online" attempts where we allow the model to specialize on the task as it goes through the evaluation dataset. Clearly showcasing the power of continual learning regime simply unavailable for any other typical LLM. Full methodology, and replication code is in the repo: https://github.com/volotat/mini-AGI Intermediate weights snapshot on Hugging Face: https://huggingface.co/Volotat/mini-AGI-undertrained

ly aligned with my interests, not some corporations. I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data. First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients. I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself. Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified. The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do. The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself. Thanks for your attention.