⚙️ Inference Engines: vLLM, Ollama, and TensorRT-LLM
Having the weights of a large LLM and a powerful server is not enough. To use AI, you need an inference engine capable of loading weights into VRAM, managing the KV Cache, and executing matrix math.
In 2026, the ecosystem has become highly specialized. Engine choice dictates tokens/s performance, initial response time (TTFT), and the ability to handle concurrent requests.
1. llama.cpp & Ollama: Kings of the workstation
Ollama has become the de facto standard for quickly testing models — as of Q4 2026, its GitHub repository exceeds 180,000 stars and its official Docker image is approaching 185 million cumulative pulls1. Under the hood, Ollama has become a dual-engine runtime: llama.cpp (C/C++) on Linux, Windows and for most architectures, and MLX (Apple), which handles compatible models by default on Apple Silicon (see 🚀 Démarrer avec Ollama).
🌟 Strengths
- Hardware versatility: Optimized to use unified memory on Mac Studio, handle offloading between RAM and GPU on modest workstations, and run on almost any CPU.
- GGUF format: Uses aggressive quantization formats (e.g.
Q4_K_M), fitting massive models into very limited VRAM without complex dependencies2. - Simplicity: A single executable, one
ollama runcommand, and an OpenAI-compatible API ready to use.
⚠️ Limits (The production wall)
The classic mistake is deploying Ollama to serve an SMB application with several simultaneous users. By default, Ollama processes only one request at a time per model (OLLAMA_NUM_PARALLEL=1) and queues the others; this parallelism can be raised, but the memory required grows with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH, and the server has neither continuous batching nor a production scheduler3. As soon as a handful of users query the same model at the same time, latency visibly degrades — that is the signal to move to vLLM.
2. vLLM: The production standard
vLLM is the reference open-source Python/C++ engine for high-throughput inference. Built for servers with NVIDIA chips (and AMD ROCm), it is designed to maximize GPU utilization under heavy load.
🌟 Strengths
- PagedAttention: vLLM popularized this technique, which manages KV Cache memory in blocks (like an OS virtual memory). This reduces memory fragmentation from ~60% to under 4% and enables massive request batching (Continuous Batching)4.
- High concurrent throughput: On multi-user architectures, vLLM can deliver overall throughput well above Ollama under concurrent load; the gap depends heavily on hardware, model and quantization — measure it with
vllm bench serveon your own prompts rather than relying on a generic factor5. - Cutting-edge format support: It handles production quantization (FP8 —
fp8_per_tensorshorthand since vLLM 0.31, the formerfp8name redirects6 —, AWQ, GPTQ) via kernels natively optimized for NVIDIA Hopper and Blackwell architectures, and natively supports Tensor Parallelism in multi-GPU setups7.
⚠️ Limits
vLLM is not designed for offloading to classic CPU RAM, nor for Apple silicon. It requires robust hardware (dedicated GPUs) and finer server parameter tuning.
3. TensorRT-LLM: Extreme NVIDIA acceleration
TensorRT-LLM is NVIDIA’s official SDK for extracting maximum physical performance from its own GPUs. Since version 1.0 (September 2025) it is built on a native PyTorch architecture, and version 1.2 (March 2026) removed the legacy TensorRT backend: there is no longer any upfront “engine” compilation, the model is served directly by trtllm-serve8.
🌟 Strengths
- Performance ceiling: It often beats all other engines on datacenter GPUs (H100, B200) thanks to techniques like Flash-Decoding.
- Native FP4: On datacenter Blackwell (B200, B300), TensorRT-LLM natively supports NVFP4 to halve VRAM footprint compared to FP8 while maintaining datacenter-class precision; on SM120 cards, support remains partial (see the limits below)8.
- Massive parallelism: It orchestrates execution graphs perfectly across multi-GPU nodes connected by NVLink.
⚠️ Limits
The constraint is no longer compilation but the release cadence: as of Q4 2026, the latest stable version is 1.3.0 (November 20, 2026), published after 30 release candidates while 1.4 was already in RC, and the 1.3 RC release notes flagged known issues on SM120 GPUs (RTX PRO 6000, RTX 50) for NVFP4 MoE and MLA models: revalidate your model on your card before making it a target89. Since 1.3, anonymous usage telemetry is enabled by default: disable it (TRTLLM_NO_USAGE_STATS=1 or --no-telemetry) on a sovereign deployment9. The learning curve remains steeper than vLLM’s.
4. SGLang: Agentic orchestration and structured generation
SGLang (Structured Generation Language) is an open-source inference engine developed by LMSys (Berkeley). Emerging as a direct competitor to vLLM in late 2023, it gained the upper hand in 2026 in two specific areas where its architecture is the most specialized: agentic loops and constrained JSON generation10.
🌟 Strengths
- RadixAttention — Shared prefix cache: SGLang organizes the KV Cache as a radix tree. When multiple requests share a long common prefix — a system prompt, retrieved RAG context, or a tool schema — that prefix is computed only once and reused by all requests that share it. In an agentic loop where the agent calls a tool, reads the result, then calls the tool again over several turns, most of the context stays identical. SGLang avoids recalculating the KV Cache on every turn, which significantly reduces TTFT on these repetitive workloads11.
- Structured JSON generation without penalty: SGLang constrains the LLM to produce output strictly conforming to a defined JSON schema, without degrading generation speed. This is a critical property for architectures where the inference engine must communicate with an application backend via typed tool calls (function calling / tool calling)10.
⚠️ Limits
- SGLang is primarily optimized for Linux + NVIDIA GPU (CUDA 13 required since 0.5.20, September 2026); AMD Instinct (MI300X → MI355X), Intel, TPU and Apple Silicon (Metal/MLX) are officially supported, with maturity varying by platform1012.
- On raw throughput benchmarks (independent requests without shared prefix), vLLM remains the reference or equivalent13.
When to choose SGLang over vLLM?
| Criterion | vLLM | SGLang |
|---|---|---|
| Raw throughput, independent requests | ✅ Reference | Comparable |
| Agentic loops, shared prefixes | ✅ Automatic Prefix Caching (on by default, hashed blocks)14 | ✅ RadixAttention (prefix tree, finer-grained sharing) |
| Constrained JSON generation | ✅ Native (xgrammar / guidance, enabled by default)15 | ✅ Native |
| Hardware compatibility (AMD, Mac) | ✅ Broad | ✅ Broad (NVIDIA CUDA 13, AMD Instinct, Intel, TPU, Apple)12 |
| Ecosystem maturity | ✅ Very broad | ✅ Mature since 2025 |
🔧 Common vLLM startup troubleshooting
The following issues are common during first vLLM installation. They occur before the server even responds to a request.
| Symptom | Probable cause | Solution |
|---|---|---|
torch.cuda.is_available() returns False | Mismatch between installed PyTorch version and system CUDA driver | Reinstall PyTorch with the matching CUDA variant: pip install torch --index-url https://download.pytorch.org/whl/cu130 (vLLM targets CUDA 13.0 by default since 0.28 and exclusively since 0.336: the driver must be compatible) |
| OOM on load — KV Cache too large | Maximum context length requested exceeds available VRAM after weight loading | Add --max-model-len 4096 (or a lower value) to vllm serve startup to reduce pre-allocated KV Cache |
| Two vLLM servers in conflict | Port 8000 already occupied by a previous instance | Add --port 8001 for the second instance; lsof -i :8000 / netstat -tulpn to identify the process occupying the port |
| Quickly test the local API | — | Use the OpenAI Python client with base_url="http://localhost:8000/v1/" and api_key="any" (without --api-key, vLLM accepts any value; --api-key does not cover all routes: a reverse proxy is mandatory in production, see Security) |
Quick test example from Python:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1/", api_key="any")response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Hello, are you working?"}], max_tokens=64,)print(response.choices[0].message.content)📋 The Architect’s Advice
For an on-premise agent project deployed at customer sites, engine choice depends purely on the architecture scenario:
- “Local Copilot” use case (one user, desktop):
Choose Ollama / llama.cpp. Native support for
GGUFmodels in Q4 on Mac or a small Windows PC delivers excellent responsiveness without server infrastructure. - “SMB Appliance” use case (10–50 users, GPU server): If the appliance is a rack server with NVIDIA GPUs (rack variant of Scenario B), switch to vLLM without hesitation; on a Mac Studio or AMD APU appliance, stay on MLX, llama.cpp or SGLang (ROCm), since vLLM is not designed for these chips. PagedAttention and continuous batching ensure the AI will not collapse when five collaborators launch RAG requests at the same time. Use weights in AWQ or FP8 precision.
- “Sovereign Datacenter” use case (high volume, multi-node):
Use TensorRT-LLM via
trtllm-serve(OpenAI-compatible API) or, if you already run Triton, via its TensorRT-LLM backend17. This is the most efficient way to amortize the cost of professional accelerators. - “Agents and backend integration” use case (tool calling, structured JSON): Prefer SGLang. Its native prefix cache management (RadixAttention) reduces latency in agentic loops, and its constrained JSON generation guarantees reliable interfaces with any application backend.
📚 Sources and References
Footnotes
-
Ollama, GitHub repository
ollama/ollama(star count) and Docker Hub imageollama/ollama(pull count), accessed 2026-10-09. https://github.com/ollama/ollama · https://hub.docker.com/r/ollama/ollama ↩ -
J. Wang et al., Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization (arXiv:2601.14277, GGUF formats), January 2026. https://arxiv.org/abs/2601.14277 ↩
-
Ollama, FAQ — How does Ollama handle concurrent requests? (
OLLAMA_NUM_PARALLEL,OLLAMA_MAX_LOADED_MODELS, request queue), accessed 2026-10-09. https://docs.ollama.com/faq ↩ -
Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). https://arxiv.org/abs/2309.06180 ↩
-
vLLM Project, Benchmarking (
vllm bench serve/latency/throughput), stable documentation accessed 2026-10-09. https://docs.vllm.ai/en/stable/benchmarking/ ↩ -
vLLM Project, Release v0.28.0 (default PyPI wheel and Docker image on CUDA 13.0,
-cu129variants), 2026-08-26 · Release v0.31.0 (quantization="fp8"renamedfp8_per_tensor, former name redirected), 2026-10-05 · Release v0.33.0 (CUDA 13.0 only wheels and images), 2026-11-02. https://github.com/vllm-project/vllm/releases/tag/v0.28.0 · https://github.com/vllm-project/vllm/releases/tag/v0.31.0 · https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3 ↩ ↩2 -
vLLM Project Documentation & Spheron Blog, vLLM Production Deployment 2026: Multi-GPU Tensor Parallel + FP8 (Model Runner V2, Hopper/Blackwell support), May 2026. https://docs.vllm.ai/en/stable/serving/parallelism_scaling/ · https://www.spheron.network/blog/vllm-production-deployment-2026/ ↩
-
NVIDIA, TensorRT-LLM Documentation & Release Notes (1.0: PyTorch by default; 1.2: TensorRT backend and
trtllm-buildremoved; SM120 known issues; Blackwell FP4), September 2026. https://nvidia.github.io/TensorRT-LLM/ · https://nvidia.github.io/TensorRT-LLM/release-notes.html · PyPItensorrt-llm(version history: 1.3.0 stable of 2026-11-20, 1.4.0rcN), accessed 2026-11-20. https://pypi.org/project/tensorrt-llm/ ↩ ↩2 ↩3 -
NVIDIA, TensorRT-LLM — Release v1.3.0 (first stable release of the 1.3 line after 30 RCs; usage telemetry enabled by default,
TRTLLM_NO_USAGE_STATS=1/DO_NOT_TRACK=1/--no-telemetryto disable it), November 20, 2026. https://github.com/NVIDIA/TensorRT-LLM/releases/tag/v1.3.0 · PyPI,tensorrt-llm(1.4.0rc0 → 1.4.0rc4, October–November 2026), checked 2026-11-19. https://pypi.org/project/tensorrt-llm/ ↩ ↩2 -
SGLang Project, SGLang — Fast Serving Framework for LLMs and VLMs (RadixAttention, structured output). https://github.com/sgl-project/sglang ↩ ↩2 ↩3
-
Lianmin Zheng et al., Efficiently Programming Large Language Models using SGLang (RadixAttention, prefix cache, TTFT reduction). https://lmsys.org/blog/2024-01-17-sglang/ ↩
-
SGLang Project, Release v0.5.20 (CUDA 12 dropped, CUDA 13 required; ROCm
gfx1151image), September 18, 2026. https://github.com/sgl-project/sglang/releases/tag/v0.5.20 ↩ ↩2 -
SGLang Contributors, SGLang vs vLLM — scaling benchmark under high concurrency (throughput comparison). https://github.com/sgl-project/sglang/issues/21061 ↩
-
vLLM Project, Automatic Prefix Caching and Engine Arguments (
enable_prefix_cachingon by default, block hashing,vllm:prefix_cache_hits/queriesmetrics), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/ · https://docs.vllm.ai/en/stable/configuration/engine_args/ ↩ -
vLLM Project, Structured Outputs (
xgrammar/guidancebackends,autoselection via--structured-outputs-config), accessed 2026-10-09. https://docs.vllm.ai/en/stable/features/structured_outputs/ ↩ -
Tenstorrent, vLLM integration with TT-Metal (fork tenstorrent/vllm, tt-metal, standard vLLM incompatibility), 2025. https://github.com/tenstorrent/tt-metal/blob/main/tech_reports/LLMs/vLLM_integration.md ↩
-
NVIDIA, TensorRT-LLM — Quick Start (
trtllm-serve, OpenAI-compatible API), accessed 2026-10-09 · Triton Inference Server,tensorrtllm_backendrepository (Triton backend for TensorRT-LLM, “PyTorch Backend (LLM API)” mode), accessed 2026-10-09. https://nvidia.github.io/TensorRT-LLM/ · https://github.com/triton-inference-server/tensorrtllm_backend ↩