Reading up on vLLM

The price of running machine‑learning intelligence is falling by orders of magnitude yearly, making AI tokens as cheap as electricity and enabling pervasive local and hosted AI use.

Speculative decoding in vLLM enables vLLM to verify multiple draft tokens per target-model pass, boosting throughput on AMD GPUs depending on drafting method and acceptance rate.

The piece outlines a six‑stage AI infrastructure journey, maps three predictable cost/latency crises, and demonstrates how ACE Gateway’s control‑plane features cut spend and prevent hardware failures at each stage.

Open-source models remain costlier to serve reliably due to infrastructure, utilization, and operational overhead, making hosted APIs from OpenAI and Anthropic more economical despite free model weights.

Netflix built an LLM serving platform using vLLM and Triton, exposing an OpenAI‑compatible API and enabling scalable constrained decoding for production workloads.

Napkin math shows serving a 32B LLM on an NVIDIA B200 GPU costs ~$9.36 per user per month when 300 users share the GPU with typical idle duty cycles.

Surya OCR 2 is a 650M-parameter vision-language model achieving 83.3% on olmOCR-bench and 87.2% on a 91-language benchmark, offering layout analysis, table recognition, and OCR in a single model.

A curated GitHub list of major CUDA programming books draws Hacker News commentary on their relevance, with many commenters recommending newer video and kernel-source materials instead.

In the May 2026 Ask HN thread, dozens of developers describe their current projects, including an AI-driven tabletop simulator, a Kagi alternative, and a FreeBSD release.

Open-weight models from China are commoditizing AI capability, breaking the monopoly moat of US closed labs, prompting protectionist moves that will harm long-term US competitiveness.

MiniMax researcher Olive Song reveals how they debug RL training, handle model hacking, and prioritize alignment and engineering discipline in open-weight model development.

Endless Terminals is an autonomous pipeline that procedurally generates terminal-use tasks without human annotation for training terminal agents with reinforcement learning.

A critique argues that AI agents don't bypass Brooks' Law, as coordination overhead and integration complexity still limit parallel agent scaling.

Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.

The article explains how prompt caching works through vLLM's paged attention and automatic prefix caching, detailing KV-cache reuse and practical tips for improving cache hits.

LLM inference works by tokenizing input, computing embeddings through transformer layers, then generating tokens autoregressively with KV caching and quantization optimizations.