vllm

Andrej Karpathy's Take: Official post on Mixtral 8x7B: https://mistral.ai/news/mixtral-of-experts/ Official PR into vLLM shows the inference code: https://github.com/vllm-project/vllm/commit/b5f882cc98e2c9c6... New HuggingFace explainer on MoE very nice: https://huggingface.co/blog/moe In naive decoding, performance of a bit above 70B (Llama 2), at inference speed of ~12.9B dense model (out of total 46.7B params). Notes: - Glad they refer to it as "open weights" release instead of "open source", which would imo require the training code, dataset and docs. - "8x7B" name is a bit misleading because it is not all 7B params that are being 8x'd, only the FeedForward blocks in the Transformer are 8x'd, everything else stays the same. Hence also why total number of params is not 56B but only 46.7B. - More confusion I see is around expert choice

Heh, so it totally depends on the use case. I use GPT4 constantly to chat through issues I am working on and get different perspectives. I cannot do that with local models. On the other hand, I have been processing a ton of text transcripts with a fine tuned llama2 13b model i've been working on, and for the tasks I have fine-tuned on, my local model is producing better results than GPT4, often taking a task that I had to do in multiple steps with GPT4, and being able to complete it in a single shot. I can run my local model through vLLM on my workstation at around the same tokens/sec as I can spend maxing out my API limits with GPT3.5-turbo (~$20/hr) while running on 2x 3090's. I'm hitting the vLLM (OpenAI clone) chat/completions endpoint. My model implements the HF chat_templates feature, and I worked on adding support for that to vLLM

Yeah, I will be soon. I was busy adding `chat template` support to vLLM recently, so the model (and any others that implement it properly) will work seamlessly with a clone of the OpenAI chat/completions endpoint. https://github.com/vllm-project/vllm/pull/1756 Now that I have that out of the way, back to model training ;).

Nothing special, other than I am running my fork of vLLM which supports HF prompt templates: https://github.com/vllm-project/vllm/pull/1365 Using --tensor-parallel-size 2 since I have 2 GPUs on my machine. Running my llama2 13b finetune for summarization and knowledge graph tasks: https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 Edit: I suppose I actually haven't documented my prompt template usage with the OpenAI API endpoint... There are some special roles that will enable some extra features with this model. Using Langchain (but works with OpenAI API directly): ChatMessagePromptTemplate.from_template(role='meta-task_name', template='summary') ChatMessagePromptTemplate.from_template(role='meta-current_date', template=datetime.now().strftime('%Y-%m-%d')) ChatMessagePromptTemplate.from_template(role='system', template='You are a hel

I wouldn’t say ROCm code is “slower”, per se, but in practice that’s how it presents. References: https://github.com/InternLM/lmdeploy https://github.com/vllm-project/vllm https://github.com/OpenNMT/CTranslate2 You know what’s missing from all of these and many more like them? Support for ROCm. This is all before you get to the really wildly performant stuff like Triton Inference Server, FasterTransformer, TensorRT-LLM, etc. ROCm is at the “get it to work stage” (see top comment, blog posts everywhere celebrating minor successes, etc). CUDA is at the “wring every last penny of performance out of this thing” stage. In terms of hardware support, I think that one is obvious. The U in CUDA originally stood for unified. Look at the list of chips supported by Nvidia drivers and CUDA releases. Literally anything from at least the past 10 years t

How does this compare to vLLM or exllama? Can it run llama2 30B on one 3090 24G or 70B on two 3090 24G? https://github.com/vllm-project/vllm https://github.com/turboderp/exllama https://github.com/turboderp/exllamav2

source code: https://github.com/vllm-project/vllm

We're finding that when running Llama-2-7B with vLLM (https://github.com/vllm-project/vllm) on an A40 GPU we're getting consistently lower time-to-first-token and lower average token generation time than GPT-3.5, even when processing multiple requests in parallel. A40s are pretty easy to get your hands on these days (much easer than A100s anyway). The 50x cheaper (that's 2% of the cost, not 50% of the cost) number does assume 100% GPU utilization, which may or may not be realistic for your use case. If you're doing batch processing as part of a data pipeline, which is not an unusual use case, you can run your GPU at 100% utilization and turn it off when the batch finishes. If you've got a highly variable workload then you're right, you'll have much lower utilization numbers. But if you work with an aggregator that can quickly hot swap LoR

lmdeploy and vllm have custom backends for Nvidia Triton Inference Server, which then actually serves up models. lmdeploy is a little more mature as it essentially uses Triton by default but I expect vllm to come along quickly as Triton Inference Server has been the "go to" for high scale and high performance serving of models for years for a variety of reasons. It appears that we'll see TensorRT-LLM integrated in Triton Inference Server directly: https://github.com/vllm-project/vllm/issues/541 How all of this will interact (vllm/lmdeploy backends + Triton + TensorRT-LLM) isn't quite clear to me right now.

I have been researching methods and projects built around training and running LLMs locally. I'm interested in what others have been using on this front (including straight up Pytorch/Transformers). Here's what I've gathered so far: Engines/APIs: - vllm: Inference and serving engine for LLMs (none quantized models only?) [1] - ollama: Go project to run, create and share LLMs [2] - llama.cpp: Inference of LLaMA models in C/C++ w/UI (including quantized models) [3] - llama-cpp-python: run OpenAI compatible API bindings for llama.cpp [4] - llm-engine: engine for fine-tuning and serving LLMs [5] - Lamini: hosted? closed source? solution for training LLMs [6] - GPT4All: free, locally run chatbot [7] - SkyPilot: framework for running LLMs, AI, and batch jobs [8] - HuggingFace Transformers: APIs and tools to download and train