text-generation-inference

I wanted to write that TGI inference engine is not Open Source anymore, but they have reverted the license back to Apache 2.0 for the new version TGI v2.0: https://github.com/huggingface/text-generation-inference/rel... Good news!

Zephyr 141B is a Mixtral 8x22B fine-tune. Here are some interesting details - Base model: Mixtral 8x22B, 8 experts, 141B total params, 35B activated params - Fine-tuned with ORPO, a new alignment algorithm with no SFT step (hence much faster than DPO/PPO) - Trained with 7K open data instances -> high-quality, synthetic, multi-turn - Apache 2 Everything is open: - Final Model: https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v... - Base Model: https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1 - Fine-tune data: https://huggingface.co/datasets/argilla/distilabel-capybara-... - Recipe/code to train the model: https://huggingface.co/datasets/argilla/distilabel-capybara-... - Open-source inference engine: https://github.com/huggingface/text-generation-inference - Open-source UI code https://github.com/huggingface/chat-ui Hav

Okay, I actually got local co-pilot set up. You will need these 4 things. 1) CodeLlama 13B or another FIM model https://huggingface.co/codellama/CodeLlama-13b-hf. You want "Fill in Middle" models because you're looking at context on both sides of your cursor. 2) HuggingFace llm-ls https://github.com/huggingface/llm-ls A large language mode Language Server (is this making sense yet) 3) HuggingFace inference framework. https://github.com/huggingface/text-generation-inference At least when I tested you couldn't use something like llama.cpp or exllama with the llm-ls, so you need to break out the heavy duty badboy HuggingFace inference server. Just config and run. Now config and run llm-ls. 4) Okay, I mean you need an editor. I just tried nvim, and this was a few weeks ago, so there may be better support. My expereicen was that is was full ho

A simple microservice would be https://github.com/huggingface/text-generation-inference . Works flawlessly in Docker on my Windows machine, which is quite shocking. Supports Mistral as well as everything else. Biggest downside is that there's no way to operate the tokenizer through the API. I put in a feature request but they said "you really ought to write your own specialized client-side code for that". Real bummer when the server already supports everything, but oh well. It has token streaming, automatically halts inference on connection close, and other niceties. Quite long startup time, but worth it as it doesn't have to be restarted with the client.

As this article is from some weeks ago and Huggingface has now implemented Paged Attention in text-generation-inference[1], I would assume the benchmark results would be quite different if done today. Would be very interesting to see more recent benchmarks if anyone has done any! [1] https://github.com/huggingface/text-generation-inference/iss...

Seems there's a fork already under the previous Apache-2.0 license by a non-hyperscaler user https://github.com/Preemo-Inc/text-generation-inference https://github.com/huggingface/text-generation-inference/iss...

This covers three things: Llama.cpp (Mac/Windows/Linux), Ollama (Mac), MLC LLM (iOS/Android) Which is not really comprehensive... If you have a linux machine with GPUs, i'd just use hugging face inference (https://github.com/huggingface/text-generation-inference). And I am sure there are other things that could be covered.

For fast inference, the HuggingFace cofounder, Thom Wolf recommends their text-generation-inference library https://github.com/huggingface/text-generation-inference https://twitter.com/Thom_Wolf/status/1664613366751526914

I tried to find the source for https://github.com/LAION-AI/Open-Assistant/blob/v0.0.1-beta2... but based on the image inspector <https://hub.docker.com/layers/ykilcher/text-generation-infer...> it seems to match up with https://github.com/huggingface/text-generation-inference/blo...