Speculative Decoding in vLLM on AMD GPUs

kept by night sep 12

Speculative decoding in vLLM enables vLLM to verify multiple draft tokens per target-model pass, boosting throughput on AMD GPUs depending on drafting method and acceptance rate.

AI At Home Part 2: Multi GPU Drifting
The author boosted Deepseek V4 Flash inference from ~10 to ~20 tokens/sec on four e-waste AMD V620 GPUs by using layer parallelism and a speculative draft model.
also on ROCm, #hardware, #open-source, #developer-tools
What every programmer should know about memory, Part 1 [LWN.net]
Ulrich Drepper’s LWN series introduces programmers to modern memory hardware, explaining DRAM, caches, NUMA, and how hardware limits affect software performance.
also on AMD, #performance, #hardware
Run AI Models Locally: A New Laptop Era Begins - IEEE Spectrum
The push to run AI models locally is driving the biggest change in laptop architecture in decades, with new NPUs, unified memory, and Microsoft's AI Foundry.
also on AMD, #hardware, #ai
The 6-Stage AI Infrastructure Journey: Navigating the Three FinOps & Hardware Crises with ACE Gateway — ACE Blog
The piece outlines a six‑stage AI infrastructure journey, maps three predictable cost/latency crises, and demonstrates how ACE Gateway’s control‑plane features cut spend and prevent hardware failures at each stage.
also on vLLM, #hardware, #ai, #developer-tools
Inference cost at scale with napkin math
Napkin math shows serving a 32B LLM on an NVIDIA B200 GPU costs ~$9.36 per user per month when 300 users share the GPU with typical idle duty cycles.
also on vLLM, #hardware, #ai