vLLM

• Spheron's like-for-like H100 runs on Llama 3.3 70B FP8 (three-engine post published March 2026, two-engine June 2026, both refreshed 19 October 2026) put SGLang roughly 1-5% ahead of vLLM on unique-prompt traffic, a gap Spheron itself calls run-to-run variance.
• Both stacks are now well out of date: they used vLLM v0.18.0 on the legacy model runner, before Model Runner V2 became the default for all dense models in v0.25.0 (11 July 2026) and for every model in v0.29.0 (9 September 2026), and SGLang v0.5.9, before breakable CUDA graphs became SGLang's default prefill path in v0.5.15.
• Pick vLLM for breadth, hardware coverage and release discipline rather than for any current throughput number.

• vllm.ai/blog/2026-05-11-vllm-tops-artificial-analysis
• vllm.ai/blog/2026-07-16-keeping-vllm-production-quality
• pytorch.org/blog/pytorch-foundation-welcomes-vllm/
• vllm.ai/blog/2025-01-27-v1-alpha-release
• github.com/sgl-project/sglang
• vllm.ai/blog/2024-09-05-perf-update
• vllm.ai/blog/2023-06-20-vllm

Vendor numbers are treated as claims until independently reproduced — how we judge. Structured benchmark comparisons are on the roadmap.

• Less of a two-horse race than it looks: in Spheron's three-engine H100 run on Llama 3.3 70B FP8, TensorRT-LLM led vLLM by 8-13% at 1-50 concurrent requests and about 16% at 100 once its engine was compiled, but it takes roughly 28 minutes to first request against about a minute for vLLM and SGLang.
• SGLang's edge is prefix-driven rather than general: with an 80% shared 512-token prefix it cut TTFT p50 by 37% at 50 concurrent requests, while on unique prompts the two sat within a few per cent, and enabling vLLM's --enable-prefix-caching narrowed the TTFT gap to roughly 15-18% at 10 concurrent requests, though at 50 Spheron still puts APC-on vLLM at 2,100 effective tok/s against SGLang's 2,550.
• Measure your own prefix overlap and re-run on current releases: those figures predate both vLLM's Model Runner V2 default (all dense models from v0.25.0, every model from v0.29.0, with v0.32.0 current) and SGLang's breakable CUDA graph default from v0.5.15.

Title

Best inference serving

Licence

Apache-2.0

Pick since

20 Jun 2023

Last reviewed

1 Nov 2026

Title holders to date

1

Official page

github.com/vllm-project/vllm