How prompt caching works - Paged Attention and Automatic Prefix Caching plus practical tips

kept by eddie dec 01, 25

The article explains how prompt caching works through vLLM's paged attention and automatic prefix caching, detailing KV-cache reuse and practical tips for improving cache hits.

Why Open-Source Models Haven't Killed the Big Dogs
Open-source models remain costlier to serve reliably due to infrastructure, utilization, and operational overhead, making hosted APIs from OpenAI and Anthropic more economical despite free model weights.
also on SGLang, vLLM, OpenAI, Anthropic, #ai-coding, #software-engineering, #developer-tools
🎙️Inside a Chinese AI Lab: How MiniMax Builds Open Models
MiniMax researcher Olive Song reveals how they debug RL training, handle model hacking, and prioritize alignment and engineering discipline in open-weight model development.
also on SGLang, vLLM, #open-source, #ai-coding
This website has been temporarily rate limited | www.warman.life
Open-weight models from China are commoditizing AI capability, breaking the monopoly moat of US closed labs, prompting protectionist moves that will harm long-term US competitiveness.
also on vLLM, OpenAI, Anthropic, #open-source
The 6-Stage AI Infrastructure Journey: Navigating the Three FinOps & Hardware Crises with ACE Gateway — ACE Blog
The piece outlines a six‑stage AI infrastructure journey, maps three predictable cost/latency crises, and demonstrates how ACE Gateway’s control‑plane features cut spend and prevent hardware failures at each stage.
also on vLLM, OpenAI, #developer-tools