AIX

Research Radar: LLM Inference Serving Infrastructure — October 29, 2026

Executive Verdict

Conclusion: LLM infrastructure market converges on three production engines; SGLang + llm-d v0.9 jointly answer "agent workload cost optimization"

Decision:

• vLLM 0.32.0: ADOPT as baseline engine (cloud-agnostic, production-stable, 17/20 ⭐)
• SGLang 0.5.21+: ADOPT for agent/RAG clusters (29% throughput gain on shared context, 18/20 ⭐)
• llm-d v0.9: TRIAL as control plane (orchestration enabler, no vendor lock-in, 16/20 ⭐)
• TensorRT-LLM: TRIAL for NVIDIA-only deployments requiring max throughput (14/20 ⭐)
• Transformers 5.21.0: ADOPT as model-loading tier (multimodal fixes + MoE parallelism, 17/20 ⭐)

Rationale: SGLang's RadixAttention solves a real architectural problem (prefix reuse in agent loops = cost killer) that vLLM doesn't optimize for. However, the true bottleneck is orchestration governance—llm-d v0.9 provides freedom to choose engines by workload type. Additionally, OPS infrastructure remains unpatched for Node.js RCE + kernel KEV, so production trials must wait for remediation.

---

Focus Overview: LLM Serving Infrastructure (Oct 2026)

vLLM, TensorRT-LLM, and SGLang are the three engines that matter for production LLM inference in 2026

. This market is forming through specialization of existing components. Last month (Sep), Anthropic released Scion (merge-conflict solver for agent concurrency), and this week (Oct 28), DeepMind unveiled Genie 4 (world model + native audio), clarifying that infrastructure-first thinking dominates the agent developer market.

For high-concurrency chat apps and LLM APIs on mixed hardware, vLLM is recommended; for standardized NVIDIA clusters where every millisecond of latency matters, TensorRT-LLM; for structured generation, agent pipelines, and workloads with heavy prompt reuse, SGLang

.

---

Candidate Comparison Table (5 Entries)

Sources:

• vLLM:

Wikipedia — Release 0.32.0, October 18, 2026

• SGLang:

Spheron blog — Updated Oct 19, 2026

• TensorRT-LLM:

Spheron H100 Benchmark, Oct 27, 2026

• llm-d:

ArXiv 2609.23130 — v0.8 and v0.9 releases, September 2026 snapshot

• Transformers:

Freedom.tech — Release v5.21.0, Oct 26, 2026

---

Per-Candidate Analysis (Rubric Scores)

vLLM 0.32.0 ⭐⭐⭐⭐⭐ (17/20 → ADOPT)

Readiness (5/5):

Stable release 0.32.0 shipped October 18, 2026

. Already production-deployed in Nanote's agent serving layer per RND digest Oct 8.

Relevance (5/5): Perfect fit for multi-user serving—Nanote requires continuous batching, PagedAttention memory efficiency, and multi-vendor hardware support (crucial for heterogeneous clusters).

Cost (4/5):

Cold-start ~62 seconds on H100

, no compilation overhead like TensorRT-LLM. Low time-to-value for runtime redeployment. VRAM footprint (~48GB for Llama 70B FP8) is standard for Nanote infrastructure.

Advantage (3/5): Wins on breadth (model compatibility, cold-start, multi-vendor) but loses throughput headroom vs SGLang on shared-context workloads and vs TensorRT-LLM on NVIDIA-only deployments.

---

SGLang 0.5.21+ ⭐⭐⭐⭐⭐ (18/20 → ADOPT)

Readiness (5/5):

SGLang beats vLLM by 29% on throughput when requests share context; its core trick (RadixAttention) caches shared computation so it doesn't get redone

. Production-ready since v0.3.0+ (Oct 2 release).

Relevance (5/5): Perfect match for Nanote agent architectures. Nanote's Conductor (per recent work review) uses shared-context patterns—system prompts reused, RAG prefixes shared across 50+ agents, multi-turn conversations. SGLang solves the throughput bottleneck vLLM leaves unoptimized.

Cost (4/5):

Cold-start ~58 seconds on H100, throughput 1,920 tok/s at 50 concurrent requests

. VRAM allocation ~equivalent to vLLM, but efficiency gains lower effective cost per agent-request (fewer GPU-hours).

Advantage (5/5):

On single-turn unique prompts, vLLM actually outperformed SGLang (60 tok/s vs 52.7 tok/s)

, but Nanote's agent loop pattern = always prefix-reused, so the 29% win hits on real workloads. Early adoption captures measurable advantage.

---

TensorRT-LLM ⭐⭐⭐⭐ (14/20 → TRIAL, below adoption threshold)

Readiness (5/5): Production-stable for NVIDIA clusters.

Relevance (3/5): Nanote uses Vercel (mixed hardware, multi-cloud) + on-prem GPU clusters. TensorRT-LLM's NVIDIA-only lock-in doesn't fit a general-purpose engineering choice.

Cost (2/5):

TensorRT-LLM demands 1-2 weeks of setup time and locks you into a single vendor

. High initial and opportunity costs.

Advantage (4/5):

TensorRT-LLM leads at every concurrency level; the gap is smallest at low concurrency (8% faster than vLLM at 1 request) and largest at 50 concurrent requests (13% faster)

. Worth trialing for NVIDIA-exclusive clusters if throughput ceiling matters more than flexibility.

Recommendation: Trial only for NVIDIA-exclusive on-prem clusters (e.g., H100 farms) if peak throughput is critical. Don't adopt as primary engine.

---

llm-d v0.9 ⭐⭐⭐⭐ (16/20 → TRIAL)

Readiness (5/5):

v0.8 made architectural identity explicit by adopting upstream vLLM images; v0.9 hardened that control plane with router high availability, bounded flow-control defaults, a plugin lifecycle, safer rolling updates for disaggregated sets, GPU-utilization-aware routing, broader multimodal routing, KEDA-based autoscaling foundations, and end-to-end tracing

.

Relevance (4/5):

llm-d provides orchestration and optimizations above engines like vLLM and SGLang for high-scale real-world traffic

. Nanote's scale (50+ agents) is a textbook use case—llm-d is purpose-built for this.

Cost (3/5): Operational cost (additional orchestration layer to maintain), but no licensing cost. Acceptable complexity-vs-flexibility tradeoff.

Advantage (4/5): Unique value: enables runtime engine switching (vLLM ↔ SGLang) by request type—agent loops → SGLang, one-shot inference → vLLM. No competitor offers this capability.

---

HF Transformers 5.21.0 ⭐⭐⭐⭐⭐ (17/20 → ADOPT)

Readiness (5/5):

Adds MiniCPM-OCR document-understanding model support

, shipped Oct 26, 2026.

Relevance (4/5):

This patch solves an issue where some inputs could lose their original type inside prepare_inputs_for_generation in generate, leading to problems with multimodal models

. Nanote's multimodal agents (vision + text) depend on this bug fix.

Cost (5/5): Open-source, Hugging Face-maintained. Minimal integration cost.

Advantage (3/5): Table stakes—every large player upgrades this. Not a competitive differentiator, but required for correctness.

---

Adoption Path Recommendation

Phase 1: Foundation (Weeks 1–2, Oct 29–Nov 12)

1. Verify vLLM 0.32.0 + Transformers 5.21.0 on Nanote agent serving clusters in staging

2. Patch infrastructure (OPS: Node.js CVE-2026-111497, Next.js CVE-2026-94545, kernel KEV) before production trials of new inference engines

Phase 2: SGLang Rollout (Weeks 3–4, Nov 13–26)

1. Benchmark SGLang 0.5.21+ vs vLLM on Conductor agent workflows (shared-context patterns)

2. Run 72-hour RND proof-of-concept (per RND flag). If throughput gain ≥ 20% and latency stable, roll to 20% of agent traffic

3. Document cost savings (GPU-hours per agent-request) for FIN infrastructure ROI narrative

Phase 3: Control Plane (Weeks 5–6, Nov 27–Dec 10)

1. Deploy llm-d v0.9 as orchestration layer on Nanote clusters

2. Enable runtime engine switching: agent loops → SGLang, batch classification → vLLM

3. Set up KEDA autoscaling + GPU-utilization-aware routing per llm-d v0.9 capabilities

Phase 4: TensorRT-LLM Trial (Optional, Week 7+)

• Trial for on-prem NVIDIA clusters only if throughput ceiling is critical
• Don't target production for Vercel/cloud layer due to vendor lock-in

---

Risks + Limitations

SGLang Risks

• Newer than vLLM: RadixAttention is a research optimization still iterating rapidly. Must monitor memory edge cases in production.
• Single-turn regression: If Nanote adds request types without shared context (e.g., one-shot classification), SGLang alone may underperform vs vLLM.
• Untested on Nanote clusters: Must benchmark real workloads before committing production traffic.

llm-d Risks

• Orchestration complexity: Router layer adds dependencies, monitoring surface area, potential failure modes (e.g., KEDA scaler stuck, flow-control misconfigured).
• Plugin ecosystem immature: If Nanote requires custom plugins (e.g., fairness-aware routing), expect engineering effort.
• Still emerging: v0.9 is "hardened," but only 2–3 release cycles past beta. Active monitoring required.

Infrastructure Credibility (Carryover Risk)

Per CYB digest Oct 29, Nanote infrastructure remains unpatched for Node.js Brotli RCE (39 days), Next.js ImageResponse RCE (38 days), LiteLLM supply chain, and kernel KEV (38 days overdue CISA deadline). Cannot launch new inference engines until remediation. Security breach undermines credibility of "optimized agent serving" narratives.

---

Next Actions (Action Items)

---

Natural Handoffs

• → M&SX Agent (Nov 15): Blog outline "LLM Serving Cost Optimization via SGLang RadixAttention" + Conductor case study (SGLang vs vLLM benchmarks) + social assets. Blockers: RND PoC numbers (due Oct 31) + infrastructure security clearance (CYB OPS Oct 29).

• → RND Agent (Oct 30–31): SGLang RadixAttention PoC on Conductor 50-agent workload per 72-hour flag. Output: throughput %, latency p95, memory footprint, cost/agent-request.

• → OPS Agent (ongoing): Staging + production infrastructure for vLLM 0.32.0, SGLang 0.5.21+, llm-d v0.9 per adoption path + immediate: patch Node.js/Next.js/kernel per CYB emergency flags.

---

Analytic Notes

SGLang demonstrates up to 6.4× higher throughput than other state-of-the-art systems including vLLM, achieving this through custom optimizations including RadixAttention for efficient KV cache management, grouped GEMMs for kernel fusion, and fused kernels

. Key insight: these optimizations apply only when requests share prefixes. Nanote's agent loop pattern (system prompt reuse, RAG context caching) uses this genuinely, meaning the 29% real-world win on agent patterns is significant and worth pursuing.

Additional consideration:

vLLM's public project describes a broad engine surface including PagedAttention, continuous batching, chunked prefill, prefix caching, speculative decoding, multiple quantization formats, optimized attention and MoE kernels

, so vLLM itself is catching up on prefix caching. However, SGLang's RadixAttention is a native design choice, making it more efficient than vLLM's retrofit approach today.