NoLiMa

But how is it even possible (bad harness?), or wise, to send 500k or 1M tokens per call? Regarding cache, how are you not hitting the 1hr cache? Also, start new chats early and often! I have been "agentic coding" since Sonnet 3.5 and after this paper came out, it became my bible: https://github.com/adobe-research/NoLiMa Last I checked, all models suck as you fill the context window. "Context engineering" is how you do this whole thing.

The benchmark GP mentioned is measuring at 128k-256k context (there's another at 524k-1024k, where 4.6 scored 78.3% and 4.7 scored 32.2%). The longer the context the worse the performance; there isn't really a qualitative step change in capability (if there is imo it happens at like 8k-16k tokens, much sooner than is relevant for multi-turn coding tasks - see e.g. this old benchmark https://github.com/adobe-research/NoLiMa ).

Thanks for the answer. OK, yes. That makes a lot more sense. I am context greedy ever since I read that Adobe research paper that I shared with you months ago. [0] The whole "context engineering" concept is certainly a thing, though I do dislike throwing around the word "engineer" all willy-nilly like that. :) In any case, thanks for the response. I just wanted to make sure that I was not missing something. [0] https://github.com/adobe-research/NoLiMa

> Over time the limitations of MCP have started to emerge. The most significant is in terms of token usage: GitHub’s official MCP on its own famously consumes tens of thousands of tokens of context, and once you’ve added a few more to that there’s precious little space left for the LLM to actually do useful work. Supabase MCP really devours your context window. IIRC, it uses 8k for its search_docs tool alone, just on load. If you actually use search_docs, it can return >30k tokens in a single reply. Workaround: I just noticed yesterday that Supabase MCP now allows you to choose which tools are available. You can turn off the docs, and other tools. [0] If you are wondering why you should care, all models get dumber as the context length increases. This happens much faster than I had expected. [1] [0] https://supabase.com/docs/guides/gettin

But Llama 4 Scout does badly on long context benchmarks despite claiming 10M. It scores 1 slot above Llama 3.1 8B in this one[1]. [1] https://github.com/adobe-research/NoLiMa

I ran NoLiMa on Quasar Alpha (GPT-4.1's stealth mode): https://news.ycombinator.com/item?id=43640166#43640790 Updated results from the authors: https://github.com/adobe-research/NoLiMa It's the best known performer on this benchmark, but still falls off quickly at even relatively modest context lengths (85% perf at 16K). (Cutting edge reasoning models like Gemini 2.5 Pro haven't been evaluated due to their cost and might outperform it.)

I only ran the benchmark on Quasar Alpha*; the rest of the scores come from the original paper [0] which was published before 3.7 was available. This is a pretty expensive benchmark to run if you're paying for API usage - I'd actually originally set out to run it on Llama 4 but abandoned that after estimating the cost. * - I also reproduced the Llama 3.1 8B result to check my setup. [0] - https://arxiv.org/abs/2502.05167 / https://github.com/adobe-research/NoLiMa

They are testing for a very straightforward needle retrieval, as LLMs traditionally were terrible for this in longer contexts. There are some more advanced tests where it's far less impressive. Just a couple of days ago Adobe released one such test- https://github.com/adobe-research/NoLiMa