RULER

Might end up being some confusion with the RULER benchmark from NVIDIA given the (somewhat shared) domain: https://github.com/NVIDIA/RULER EDIT: by shared I only mean the adjacency to LLMs/AI/ML, RL is a pretty big differentiator though and project looks great

There's the nominal context length, and the effective one. You need a benchmark like the needle-in-a-haystack or RULER to determine the latter. https://github.com/NVIDIA/RULER

https://github.com/NVIDIA/RULER results in benchmark other than needle in haystack seem solid all the way to 128k

It should be benchmarked against something like RULER[1] 1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)

Absolutely. Gemini results tend to drop off after 128k tokens according to RULER: https://github.com/hsiehjackson/RULER

The needle in the haystack test gives a very limited view of the model’s actual long context capabilities. It’s mostly used because early models were terrible at it and it’s easy to test. In fact, most recent models now do pretty good at this one task, but in practice, their ability to do anything complex drops off hugely after 32K tokens. RULER is a much better test: https://github.com/hsiehjackson/RULER > Despite achieving nearly perfect performance on the vanilla needle-in-a-haystack (NIAH) test, all models (except for Gemini-1.5-pro) exhibit large degradation on tasks in RULER as sequence length increases. > While all models claim context size of 32k tokens or greater (except for Llama3), only half of them can effectively handle sequence length of 32K by exceeding a qualitative threshold, Llama2-7b performance at 4K (85.6%). The perfo

Remember when they claimed Yi had 200k context length despite it having 16k of usable context? I remember, because I spent non-trivial effort trying to make it work for long-form technical summarization. My lackluster findings were validated by RULER. https://github.com/hsiehjackson/RULER

Can someone test this with ruler please? https://github.com/hsiehjackson/RULER In practice all of these long contexts show degraded performance (there's a table on the repo). For my NLP work I find that GPT-4-turbo is much worse after 32k-ish.

We got RULER 3 days ago. https://arxiv.org/abs/2404.06654 https://github.com/hsiehjackson/RULER