Reading up on ai-inference
5 deep · digging since feb 16
- A Guide to AI Inference Engineering - ByteByteGo Newsletter
LLM inference splits into compute-bound prefill and memory-bound decode, driving optimization techniques like batching, quantization, speculative decoding, and disaggregation.
- Apple Silicon costs more than OpenRouter
Running local LLMs on Apple Silicon costs about 3x more per million tokens than using OpenRouter, and also runs slower.
- Greetings, Earthlings: Philip Johnston of Starcloud on Data Centers in Space
Falling launch costs and rising terrestrial constraints will make space-based AI data centers cheaper than Earth-based ones within a decade, potentially creating a trillion-dollar annual CapEx market for inference workloads.
- The emerging role of SRAM-centric chips in AI inference
SRAM-centric chips outperform GPUs on memory-bound AI inference workloads, particularly decode, by placing memory near compute for faster bandwidth.
- Two different tricks for fast LLM inference
A speculative analysis of OpenAI's and Anthropic's fast inference strategies, claiming batch-size optimization and Cerebras hardware enable speed at the cost of model quality, though experts challenge key technical assumptions.