pyllms

The library they (probably) use for abstracting the interfaces of different LLMs they use is also open [1]: > PyLLMs is a minimal Python library to connect to various Language Models (LLMs) with a built-in model performance benchmark. https://github.com/kagisearch/pyllms

Sharing PyLLMs [1] reasoning benchmark results for some of the recent models. Surprised by nemo (speed/quality) and mistral large is actually pretty good (but painfully slow). AnthropicProvider('claude-3-haiku-20240307') Median Latency: 1.61 | Aggregated speed: 122.50 | Accuracy: 44.44% MistralProvider('open-mistral-nemo') Median Latency: 1.37 | Aggregated speed: 100.37 | Accuracy: 51.85% OpenAIProvider('gpt-4o-mini') Median Latency: 2.13 | Aggregated speed: 67.59 | Accuracy: 59.26% MistralProvider('mistral-large-latest') Median Latency: 10.18 | Aggregated speed: 18.64 | Accuracy: 62.96% AnthropicProvider('claude-3-5-sonnet-20240620') Median Latency: 3.61 | Aggregated speed: 59.70 | Accuracy: 62.96% OpenAIProvider('gpt-4o') Median Latency: 3.25 | Aggregated speed: 53.75 | Accuracy: 74.07% | [1] https://github.com/kagisearch/pyllms

Based on PyLLMs benchmark. [1] Slightly better than Haiku and slightly slower. Much cheaper. OpenAIProvider('gpt-4o-mini') Total Cost: 0.00385 | Aggregated speed: 105.72 tok/sec | Accuracy: 51.85% AnthropicProvider('claude-3-haiku-20240307') Total Cost: 0.00735 | Aggregated speed: 117.53 tok/sec | Accuracy: 48.15% [1] https://github.com/kagisearch/pyllms

Tried it with a few PyLLMs benchmarks [1] and it did not do very well. Q. Use m to substitute p, a to substitute e, n to substitute a, g to substitute c, o to substitute h, how to spell peach under this rule? A. m: p a: e n: a g: c o: h how: peach Q. write three sentences each ending with the word apple A. 1. The apple tree was covered in apples. 2. She sliced the apple with a knife. 3. The apple juice was too sweet. Wonder if the synthetic benchmarks are very biased and not that useful. [1] https://github.com/kagisearch/pyllms/blob/6c11cd51b099d33e61...

For anyone reading this, these are the actual prompts being used to assess the models. https://github.com/kagisearch/pyllms/blob/ca9ad4d4bfdd9d58fe...

Less scientific, but arguably more practical benchmarks here: https://github.com/kagisearch/pyllms#model-benchmarks

Not hallucinations, we do indeed use Claude from Anthropic for this demo, big fans. The entire demo is 20 lines of code, and uses PyLLMs [1] [1] https://github.com/kagisearch/pyllms

Hi HN, We needed a simple way to connect to the top AI models to experiment, prototype and evaluate them. Main features: - Connect to top LLMs in few lines of code (currenly OpenAI, Anthropic and AI21 are supported) - Response meta includes tokens processed, cost and latency standardized across the models - Multi-model support: Get completitions from different models at the same time - LLM benchmark: Eevaluate models on quality, speed and cost The benchmark uses predefine questions to test AI reasoning abilities across a range of "hard" queries. The outputs are then automatically evaulauted using a powerful model (gpt-4 recommended): https://github.com/kagisearch/pyllms/blob/990855968b4bc26ab6... This helped uncover a hidden gem among models: 'claude-instant-v1' which is 4x faster, 2x cheaper and similar quality to 'crowd favorite' gpt-3.

claude-instant-v1 is one of the "best kept secrets". It is comperable in quality to gpt-3.5-turno, while being four times faster (!) and at half the price (!). We just released a minimal python library PyLLMs [1] to simplify using various LLMs (openai, anthropic, AI21..) and as a part of that we designed a LLM benchmark. All open source. [1] https://github.com/kagisearch/pyllms/tree/main#benchmarks