ChainForge

"WebNN: Web Neural Network API" https://news.ycombinator.com/item?id=36158663 : > - Src: https://github.com/webmachinelearning/webnn W3C Candidate Recommendation Draft: > - Spec: https://www.w3.org/TR/webnn/ > WebNN API: https://www.w3.org/TR/webnn/#api : >> 7.1. The `navigator.ml` interface >> webnn-polyfill E.g. Promptfoo, ChainForge, and LocalAI all have abstractions over many models; also re: Google Desktop and GNU Tracker and NVIDIA's pdfgpt: https://news.ycombinator.com/item?id=39363115 promptfoo: https://github.com/promptfoo/promptfoo ChainForge: https://github.com/ianarawjo/ChainForge LocalAI: https://github.com/go-skynet/LocalAI

This seems like a sheets implementation of something like ChainForge (https://github.com/ianarawjo/ChainForge). It's curious that Anthropic is entering the LLMOps tooling space ---this definitely comes as a surprise to me, as both OpenAI and HuggingFace seem to avoid building prompt engineering tooling themselves. Is this a business strategy of Anthropic's? An experiment? Regardless, it's cool to see a company like them throw their hat into the LLMOps space beyond being a model provider. Interested to see what comes next.

This seems like a sheets implementation of something like ChainForge (https://github.com/ianarawjo/ChainForge). It's curious that Anthropic is entering the LLMOps tooling space ---this definitely comes as a surprise to me, as both OpenAI and HuggingFace seem to avoid building prompt engineering tooling themselves. Is this a business strategy of Anthropic's? An experiment? Regardless, it's very cool to see a company like them throw their hat into the LLMOps space beyond being a model provider. Interested to see what comes next.

ChainForge lets you do this, and also setup ad-hoc evaluations with code, LLM scorers, etc. It also shows model responses side-by-side for the same prompt: https://github.com/ianarawjo/ChainForge

ChainForge has similar functionality for comparing : https://github.com/ianarawjo/ChainForge LocalAI creates a GPT-compatible HTTP API for local LLMs: https://github.com/go-skynet/LocalAI Is it necessary to have an HTTP API for each model in a comparative study?

Looks nice, and it's nice that it also supports function call simulation. I've been collecting a list of tools for prompt engineering, I've added Knit now. Newly added: https://promptknit.com/ Newly added: https://github.com/promptfoo/promptfoo Newly added: https://promptlayer.com https://promptable.ai/ https://github.com/ianarawjo/ChainForge https://trywale.com/ https://promptknit.com/ https://promptrefine.com/ https://www.vellum.ai/ https://promptclub.ai/ https://www.everyprompt.com/ https://scale.com/spellbook

I think you should probably mention that its source is available! [0] I don't personally have a need for this right now, but I can really see the use for the parameterised queries, as well as comparisons across models. Thanks for your efforts! 0: https://github.com/ianarawjo/ChainForge

I like the support for Vector DBs and LLaMa-2. I'm curious as to whether and what influences compelled PromptTools, and how it differs from other tools in this space. For context, we've also released a prompt engineering IDE, ChainForge, which is open-source and has many of the features here, such as querying multiple models at once, prompt templating, evaluating responses with Python/JS code and LLM scorers, plotting responses, etc (https://github.com/ianarawjo/ChainForge and a playground at http://chainforge.ai). One big problem we're seeing in this space is over-trust in LLM scorers as 'evaluators'. I've personally seen that minor tweaks to a scoring prompt can sometimes result in vastly different evaluation 'results.' Given recent debacles (https://news.ycombinator.com/item?id=36370685), I'm wondering how we can design LLMOps tools fo

No problem! I guess I will make a plug myself --we've been working on a similar 'prompt engineering' tool, ChainForge (https://github.com/ianarawjo/ChainForge). It's targeted towards slightly different users and use cases than promptfoo --geared more towards early-stage, 'quick-and-dirty' explorations of differences between prompts and models for less experienced programmers, versus the kind of continuous benchmarking and verification testing power that promptfoo offers. I particularly like promptfoo's support for CI, which I haven't seen anywhere else, and is very important for developers pushing prompts into production (esp since OpenAI keeps updating their models every few months...).

Hey all! I've been developing a prompt engineering interface that helps users query LLMs with parametrized prompts and compare responses across models. It's an early demo, but already we've used it internally in my academic lab to evaluate and choose prompts for other research projects that involve building LLM applications. Let me know what you think, or if you encounter any bugs or issues. Blog post here: https://ianarawjo.medium.com/introducing-chainforge-a-visual...