BAbavell·Dec 27, 2024
8x H200s recommended:
https://github.com/sgl-project/sglang/tree/main/benchmark/de...
In DeepSeek-V3sgl-project
sglang
Python2024–2026
sgl-project / sglang
sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
13Distinct people
14Mentions
0Explicit recommenders
38.8KGitHub stars
What to know before exploring
What it is
SGLang is a high-performance serving framework for large language models and multimodal models.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 2:59:58 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year14 dated mentions
Observed mentions: 2024-02-24 00:22:34 UTC to 2026-10-15 19:08:28 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 6 | 0 | 6 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 5 | 0 | 5 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 3 | 1 | 2 |
What Hacker News says
6 sourcesIt has a clever way to decode multiple valid tokens at once, rather than just one token at a time.
Corresponding project link: https://github.com/sgl-project/sglang
In .txt raises $11.9M to make language models programmablePretty sure this was never questioned for batched requests, sg-lang/lmdeploy/tensorRT-LLM will have nearly twice as reported speeds with INT8 (fp16 A100 benched here https://github.com/sgl-project/sglang?tab=readme-ov-file#ben...)
In Benchmarks show even an old Nvidia RTX 3090 is enough to serve LLMs to thousandsI was initially impressed with the landing page, but it does look a bit suspect when things are claimed to be 100x faster without much info on the HW acceleration or the model sizes.
My best guess is that they're using two approaches to get this running faster:
- structured generation techniques from sglang (https://github.com/sgl-project/sglang) that allow them to generate faster JSON (with look-ahead / pre-fill) with strong guarantees on the output (i.e. 100% reliable, without requiring any retries).
- distilling a gpt-3.5 turbo-esque model from GPT-4 JSON outputs, and using it in conjuction with above to give the additional performance boosts on inference.
It doesn't seem like they're deploying on any custom silicon, nor have they optimized GPU kernels to suggest that the speed ups came there.
In Inversion: Fast, Reliable Structured LLMs> The most obvious usage of this is forcing a model to output valid JSON
Isn't this something that Outlines [0], Guidance [1] and others [2] already solved much more elegantly?
0. https://github.com/outlines-dev/outlines
1. https://github.com/guidance-ai/guidance
2. https://github.com/sgl-project/sglang
In Show HN: Prompts as WASM ProgramsThanks! Again, partnerships over customers. If you're experienced and have the technical chops to make a MI300x sing, we want to work with you. Our model is that we are the capex/opex investor for businesses. As much as I love software, Hot Aisle is more of a hardware business. Running super high end large scale compute is an extreme challenge in itself. We are less interested in building the software side of things and want to foster those who can focus on that side.
https://github.com/unslothai/unsloth/issues/160
https://github.com/search?q=repo%3Apredibase%2Florax+rocm&ty...
https://github.com/sgl-project/sglang/issues/157
https://github.com/casper-hansen/AutoAWQ (supports rocm)
In AMD ROCm Software Blogs