vllm

How do you prompt the model? In my experience, Qwen3-VL models have very accurate grounding capabilities (I’ve tested Qwen3-VL-30B-A3B-Instruct, Qwen3-VL-30B-A3B-Thinking, and Qwen3-VL-235B-A22B-Thinking-FP8). Note that the returned values are not direct pixel coordinates. Instead, they are normalized to a 0–1000 range. For example, if you ask for a bounding box, the model might output: ```json [ {"bbox_2d": [217, 112, 920, 956], "label": "cat"} ] ``` Here, the values represent [x_min, y_min, x_max, y_max]. To convert these to pixel coordinates, use: [x_min / 1000 * image_width, y_min / 1000 * image_height, x_max / 1000 * image_width, y_max / 1000 * image_height] Also, if you’re running the model with vLLM > 0.11.0, you might be hitting this bug: https://github.com/vllm-project/vllm/issues/29595

I trained the agent itself to write SQL, run it, check results, then rewrite until correct. The write and rewrite policies are optimized with RL, using a client–server setup in Agent Lightning and a LangGraph state machine. On a 500-sample Spider eval subset, Qwen2.5-Coder-3B with 4096 context reaches 80.4% at three turns of write and rewrite, 80.2% at one turn. After training, model Qwen2.5-Coder-1.5B can be even better than Qwen2.5-Coder-3B (untrained). I have compared multiple models and settings, hoping to shed light on tuning AI agents. Full article: https://medium.com/@yugez/training-ai-agents-to-write-and-se... Related projects: - Agent Lightning as the glue: https://github.com/microsoft/agent-lightning - verl for RL algorithms: https://github.com/volcengine/verl - vLLM for efficient rollouts: https://github.com/vllm-project/vllm -

Your reply adds more confusion, imo. The inference code and model architecture IS open source[0] and there are many other high quality open source implementations of the model (in many cases contributed by Google engineers[1]). To your point: they do not publish the data used to train the model so you can't re-create it from scratch. [0] https://github.com/google-deepmind/gemma [1] https://github.com/vllm-project/vllm/pull/2964

After seeing the Docker image for vllm jump +5Gb (to 10Gb!) over the past five months, I grew suspicious of vllm's development practices [1]. It's not easy, for sure, to deal with all those flaky python modules [2]. But having the CUDA packages four times in different layers is questionable! [3] Yet again, as a college mate of mine used to say, "Don't change it. It works." -- [1]: https://hub.docker.com/r/vllm/vllm-openai/tags [2]: https://github.com/vllm-project/vllm/issues/13306 [3]: These kinds of workarounds tend to end up accumulating and never get reviewed back: - https://github.com/vllm-project/vllm/commit/b07d741661570ef1... - https://github.com/vllm-project/vllm/commit/68d37809b9b52f4d... (this one in particular probably accounts for +3Gb)

No, vLLM is a thing for serving language models: https://github.com/vllm-project/vllm

While Llama 4 had a pretty bad launch (the LM Arena gaming in particular is terrible), having run my own evals on it (using the April 5 v0.8.3 vLLM release - https://blog.vllm.ai/2025/04/05/llama4.html , so before the QKNorm fix https://github.com/vllm-project/vllm/pull/16311) - it seemed pretty decent to me. For English, on a combination of MixEval, LiveBench, IFEval, and EvalPlus Maverick FP8 (17B/400B) was about on par with DeepSeek V3 FP8 (37B/671B) and Scout (17B/109B) was punching in the ballpark of Gemma 3 27B, but not too far off Llama 3.3 70B and Mistral Large 2411 (123B). Llama 4 claimed to be trained on 10X more multilingual tokens than Llama 3 and testing on Japanese (including with some new, currently unreleased evals) the models did perform better than Llama 3 (although I'd characterize their overall Japanese performance as

There were actually multiple bugs which impacted long context benchmarks and general inference - I helped fix some of them. 1. RMS norm eps was 1e-6, but should be 1e-5 - see https://github.com/huggingface/transformers/pull/37418 2. Llama 4 Scout changed RoPE settings after release - conversion script for llama.cpp had to be fixed. See https://github.com/ggml-org/llama.cpp/pull/12889 3. vLLM and the Llama 4 team found QK Norm was normalizing across entire Q & K which was wrong - accuracy increased by 2%. See https://github.com/vllm-project/vllm/pull/16311 If you see https://x.com/WolframRvnwlf/status/1909735579564331016 - the GGUFs I uploaded for Scout actually did better than inference providers by +~5% on MMLU Pro. https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-... has more details

FWIW I'm getting 29 TPS on Ollama on my 7900 XTX with the 27b qat. You can't really compare inference engine to inference engine without keeping the hardware and model fixed. Unfortunately Ollama and vLLM are therefore incomparable at the moment, because vLLM does not support these models yet. https://github.com/vllm-project/vllm/issues/16856

vLLM supports MLA for Deepseek models as of 3 weeks ago. 3x higher generation throughput and 10x token memory capacity. https://github.com/vllm-project/vllm/releases/tag/v0.7.1 MHA is still faster in low QPS regime apparently. https://neuralmagic.com/blog/enhancing-deepseek-models-with-... Also published this month was theoretical proof showing that for the same KV Cache overhead, MLA consistently offers greater expressive power than GQA. Furthermore, widely used GQA-based pre-trained models (e.g. LLaMA, Qwen, Mixtral) can be converted into MLA-based models. https://arxiv.org/pdf/2502.07864