FastChat

nice work IF you want free tokens and IF your visitors agree for all their input and output to be used by all AIs involved for research and training or whatever.. you might be able to strike up the same kind of dealio that LMSYS has with lmarena - > github.com/lm-sys/FastChat - > lmarena .ai to use the site you have a big obvious popup explaining the data use quickly and efficiently

Hey, Simon! Have you ever considered to host private evals? I think, with the weight of the community behind you, you could easily accumulate a bunch of really high-quality, "curated" data, if you will. That is to say, people would happily send it to you. More people should self-host stuff like https://github.com/lm-sys/FastChat without revealing their dataset, I think, and we would probably trust it more than the public stuff, considering they already trust _you_ to some extent! So far the private eval scene is just a handful of guys on twitter reporting their findings in unsystematic manner, but a real grassroots approach backed up by a respectable influencer would go a long way to change that. Food for thought.

So far, non-English inputs have been most telling: I deal with Ukrainian datasets mostly, and what we see is OpenAI models, the Chinese models, of course, and Llama's, to admittedly, lesser extent—all degrading disproportionately compared to the other models. You know what model degrades the least comparatively? Gemma 27b. The arena numbers would suggest it's not so strong, but they'd actually managed to make something useful for the whole world (I can only judge re: Ukrainian, of course, but I suspect it's probably equally good in the other languages, too.) However, nothing can compete currently with Sonnet 3.5 in reasoning. I predict a surge in the private eval scene when people inevitably grow wary of leaderboard-propaganda. More people should host https://github.com/lm-sys/FastChat

gpt2-chatbot is not the only "mystery model" on LMSYS. Another is "deluxe-chat". When asked about it in October last year, LMSYS replied [0] "It is an experiment we are running currently. More details will be revealed later" One distinguishing feature of "deluxe-chat": although it gives high quality answers, it is very slow, so slow that the arena displays a warning whenever it is chosen as one of the competitors [0] https://github.com/lm-sys/FastChat/issues/2527

How does it compare with something like FastChat? https://github.com/lm-sys/FastChat Feature set seems like a decent amount of overlap. One limitation of FastChat, as far as I can tell, is that one is limited to the models that FastChat supports (though I think it would be minor to modify it to support arbitrary models?)

Looks like the Vicuna repo is Apache 2.0 also[1]. What's the interpretation of copyright law that would prevent the code being Apache 2.0 based on the source of the fine-tuning dataset? [1] https://github.com/lm-sys/FastChat

For something a little heavier but much more robust in terms of features/functionality I've been enjoying FastChat: https://github.com/lm-sys/FastChat It allows you to plug in different backends so that you can use OpenAI compatible clients with various LLM's, selfhosted or otherwise.

Any reason you're doing that vs. using Lambda Labs / Replicate / together.ai / Banana.dev, etc. There's a lot of good model deployment platforms that would make it easy to call your model behind a hosted endpoint -- If you do want to self-host - there's some great libraries like https://github.com/lm-sys/FastChat and https://github.com/ggerganov/llama.cpp that might be helpful If none of these really solve your issue - feel free to email me and I'm happy to help you figure something out - krrish@berri.ai

There is probably use for go-skynet/LocalAI[0] or lm-sys/FastChat[1] which can emulate an OpenAI API using local models. 0: https://github.com/go-skynet/LocalAI 1: https://github.com/lm-sys/FastChat/ Edit: idk if any of this support function calling tho

Cool stuff! How does this compare with Fastchat, which seems like another open source project that helps run LLM models? At a glance, it seems like it's going for lots of similar goals (run LLMs with interoperable APIs): https://github.com/lm-sys/FastChat

https://github.com/lm-sys/FastChat The hardest part is downloading the 30GB of weights.