pyllms

Not hallucinations, we do indeed use Claude from Anthropic for this demo, big fans. The entire demo is 20 lines of code, and uses PyLLMs [1] [1] https://github.com/kagisearch/pyllms

Hi HN, We needed a simple way to connect to the top AI models to experiment, prototype and evaluate them. Main features: - Connect to top LLMs in few lines of code (currenly OpenAI, Anthropic and AI21 are supported) - Response meta includes tokens processed, cost and latency standardized across the models - Multi-model support: Get completitions from different models at the same time - LLM benchmark: Eevaluate models on quality, speed and cost The benchmark uses predefine questions to test AI reasoning abilities across a range of "hard" queries. The outputs are then automatically evaulauted using a powerful model (gpt-4 recommended): https://github.com/kagisearch/pyllms/blob/990855968b4bc26ab6... This helped uncover a hidden gem among models: 'claude-instant-v1' which is 4x faster, 2x cheaper and similar quality to 'crowd favorite' gpt-3.

claude-instant-v1 is one of the "best kept secrets". It is comperable in quality to gpt-3.5-turno, while being four times faster (!) and at half the price (!). We just released a minimal python library PyLLMs [1] to simplify using various LLMs (openai, anthropic, AI21..) and as a part of that we designed a LLM benchmark. All open source. [1] https://github.com/kagisearch/pyllms/tree/main#benchmarks