raink

The open source ranking library is really interesting. It's using a type of merge sort where the comparator function is an llm comparing (but doing batches >2 for fewer calls). Reducing problems to document ranking is effectively a type of test-time search - also very interesting! I wonder if this approach could be combined with GRPO to create more efficient chain of thought search... https://github.com/BishopFox/raink?tab=readme-ov-file#descri...

I published a nearly identical tool, referencing the same paper, a few weeks ago :) Although I implemented a listwise algorithm instead of pairwise as described in the paper; ends up being a lot faster. https://github.com/BishopFox/raink https://bishopfox.com/blog/raink-llms-document-ranking raink \ -f testdata/sentences.txt \ -r 10 \ -s 10 \ -p 'Rank each of these items according to their relevancy to the concept of "time".' | jq -r '.[:10] | map(.value)[]' | nl 1 The train arrived exactly on time. 2 The old clock chimed twelve times. 3 The clock ticked steadily on the wall. 4 The bell rang, signaling the end of class. 5 The rooster crowed at the break of dawn. 6 She climbed to the top of the hill to watch the sunset. 7 He wat

I think a lot of AI-augmented security problems can be decomposed into "show me the best thing in this list of things": - the changed function in a patch diff that most closely relates to a given security advisory - the injection point in a webapp that seems most likely to cause a state change on the backend - the static code analyzer result that would have most severe impact if a sink were actually reachable It's notoriously difficult to get an LLM to seriously consider all items when presented with a big list of input—so I built raink, a CLI tool to harness LLMs for general purpose document ranking. Blog post here: https://bishopfox.com/blog/raink-llms-document-ranking