Engram

Nice work thanks for doing it, a couple of notes: It looks like you are referencing Engram [1], but aren't actually gathering a n-gram (e.g. n=1) but rather individual token_ids. use_cache=True in ``` with torch.no_grad(): outputs = model.generate(*inputs, max_new_tokens=150, do_sample=False, pad_token_id=tokenizer.eos_token_id, use_cache=True) return tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True).strip() ``` I think there's a bug around not updating current_train_input_ids as more tokens are updated and processed. To be honest I don't fully understand the code so I could be wrong. Happy to chat more if you're interested I'll shoot you an email! Lastly just for my sake, please correct me if I am wrong, but my reading is that you are learning an additional gate on top of a select number of layers that

Please don't slander the most open AI company in the world. Even more open than some non-profit labs from universities. DeepSeek is famous for publishing everything. They might take a bit to publish source code but it's almost always there. And their papers are extremely pro-social to help the broader open AI community. This is why they struggle getting funded because investors hate openness. And in China they struggle against the political and hiring power of the big tech companies. Just this week they published a serious foundational library for LLMs https://github.com/deepseek-ai/TileKernels Others worth mentioning: https://github.com/deepseek-ai/DeepGEMM a competitive foundational library https://github.com/deepseek-ai/Engram https://github.com/deepseek-ai/DeepSeek-V3 https://github.com/deepseek-ai/DeepSeek-R1 https://github.com/deeps

Different architectures, different RL training loops, maybe memory modules [1][2] as part of the architecture, focusing on efficiency, the giant troves of data we're generating by using claude code/gemini-cli/opencode, there's lots of research to be made. [1] https://research.google/blog/titans-miras-helping-ai-have-lo... [2] https://github.com/deepseek-ai/Engram