PEpeter_d_sherman·Oct 15, 2026Submission
SGLang – High-performance serving framework for LLM's and multimodal models
1 points · Open discussionsgl-project
sglang
Python2024–2026
sgl-project / sglang
sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
13Distinct people
14Mentions
0Explicit recommenders
38.8KGitHub stars
What to know before exploring
What it is
SGLang is a high-performance serving framework for large language models and multimodal models.
GitHub
Where it is used
No documented product use was verified in the last source check.
What people are saying
No recommendation phrase was found in the indexed comments.
No classified warning is attached to this page. Read the discussions before treating silence as approval.
Also in these discussions
No other repository is repeatedly linked in the same discussions.
This week on Hacker News
Nov 8–Nov 15 UTC · Direct repository links and separately labeled related project stories. Indexed through 11/15/2026, 3:38:16 PM UTC.
No direct repository links were found in the indexed HN discussions over the past seven days. Older discussions are available below.
Mentions over time
HN mentions by year14 dated mentions
Observed mentions: 2024-02-24 00:22:34 UTC to 2026-10-15 19:08:28 UTC.
* An incomplete calendar period is marked with an outlined bar. Its count is not directly comparable with a full period.
Counts come from the repository’s dated source evidence. Index cutoff: 2026-11-02 08:16:59 UTC.
View mention counts and date ranges
| Period | Start (UTC) | End (exclusive, UTC) | Mentions | Submissions | Comments |
|---|---|---|---|---|---|
| 2024 | 2024-01-01 00:00:00 UTC | 2025-01-01 00:00:00 UTC | 6 | 0 | 6 |
| 2025 | 2025-01-01 00:00:00 UTC | 2026-01-01 00:00:00 UTC | 5 | 0 | 5 |
| 2026 (partial) | 2026-01-01 00:00:00 UTC | 2027-01-01 00:00:00 UTC | 3 | 1 | 2 |
What Hacker News says
3 sourcesAuthor here. We serve several tenants on shared SGLang nodes, and our p99 TTFT was bad while the median looked fine. The cause was head-of-line blocking in prefill. SGLang's scheduler lets only one request be mid-chunk at a time, and that request takes the whole chunked-prefill budget every round. So one cold 600K-token prompt held the prefill slot for about 105s. Everything queued behind it waited too, including requests with a 99% prefix-cache hit that had almost nothing left to prefill.
The fix turns that single slot into a set, capped per request by long_prefill_token_threshold. Most of the work was in the places that assumed only one partial prefill could exist: KV reservation, output streaming, parking chunks under memory pressure, and spec-decode tail tokens.
In that scenario the big request slowed from 105s to 112s. The six reques
Read the rest
ts behind it went from about 105s to between 0.8s and 2.4s.
It isn't free. If every request is huge, FCFS is already the right policy and chunking only adds overhead. A lone long prefill still pays the per-round cap even when nothing else wants the budget. Peak KV usage also rises somewhat. The threshold is a ceiling today, and I'd rather it were a floor, so concurrency adapts at runtime without tuning.
Upstream PR: https://github.com/sgl-project/sglang/pull/34623
I'm happy to go into scheduler details. I'd especially like to hear from anyone who has handled this differently, for example with prefill/decode disaggregation or priority queues.
There are several things at play:
Inference stack efficiency: Many of these providers take off the shelf sglang / vllm / trtllm and hope for the best. Meanwhile DeepSeek team is known for pushing the boundary of optimizations.
Now, sglang and vllm are great pieces of software, but take DeepSeek's Sparse Attention (DSA). Introduced 1.5 years ago (https://arxiv.org/abs/2512.02556), used by DeepSeek 3.2, GLM 5, DeepSeek V4. Only now is it slowly strating to get optimized in the major inference engines: (https://github.com/sgl-project/sglang/issues/19380 https://github.com/sgl-project/sglang/pull/22851 etc.). Of course, DS V4 adds extra optimizations into the model architecture on top of DSA, and those will take more time to be taken full advantage of by the open source inference engines.
Privacy: Betting that people will pay extra for infere
Read the rest
nce hosted outside China. This is especially true with DeepSeek, because DeepSeek is transparent about using API data for model improvements.
And few other things (scale (matters a lot for MoEs), reliability, soft enterprise lock in, etc.)
---
There is also, likely, tacit collusion at play here. Look at GLM 5 and GLM 5.1 prices. GLM 5 and 5.1 cost the same to run, but providers decided to charge much more for 5.1 because it is much better model, and because Z.AI raised their price as well.