Inference cost at scale with napkin math

kept by eddie jun 15

Napkin math shows serving a 32B LLM on an NVIDIA B200 GPU costs ~$9.36 per user per month when 300 users share the GPU with typical idle duty cycles.

The 6-Stage AI Infrastructure Journey: Navigating the Three FinOps & Hardware Crises with ACE Gateway — ACE Blog
The piece outlines a six‑stage AI infrastructure journey, maps three predictable cost/latency crises, and demonstrates how ACE Gateway’s control‑plane features cut spend and prevent hardware failures at each stage.
also on vLLM, #infrastructure, #hardware, #ai
datalab-to/surya-ocr-2
Surya OCR 2 is a 650M-parameter vision-language model achieving 83.3% on olmOCR-bench and 87.2% on a 91-language benchmark, offering layout analysis, table recognition, and OCR in a single model.
also on vLLM, #machine-learning, #ai