Reading up on vLLM

Napkin math shows serving a 32B LLM on an NVIDIA B200 GPU costs ~$9.36 per user per month when 300 users share the GPU with typical idle duty cycles.

Surya OCR 2 is a 650M-parameter vision-language model achieving 83.3% on olmOCR-bench and 87.2% on a 91-language benchmark, offering layout analysis, table recognition, and OCR in a single model.

A curated GitHub list of major CUDA programming books draws Hacker News commentary on their relevance, with many commenters recommending newer video and kernel-source materials instead.

In the May 2026 Ask HN thread, dozens of developers describe their current projects, including an AI-driven tabletop simulator, a Kagi alternative, and a FreeBSD release.

Open-weight models from China are commoditizing AI capability, breaking the monopoly moat of US closed labs, prompting protectionist moves that will harm long-term US competitiveness.

MiniMax researcher Olive Song reveals how they debug RL training, handle model hacking, and prioritize alignment and engineering discipline in open-weight model development.

Endless Terminals is an autonomous pipeline that procedurally generates terminal-use tasks without human annotation for training terminal agents with reinforcement learning.

A critique argues that AI agents don't bypass Brooks' Law, as coordination overhead and integration complexity still limit parallel agent scaling.

Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.

The article explains how prompt caching works through vLLM's paged attention and automatic prefix caching, detailing KV-cache reuse and practical tips for improving cache hits.

LLM inference works by tokenizing input, computing embeddings through transformer layers, then generating tokens autoregressively with KV caching and quantization optimizations.