A Guide to AI Inference Engineering - ByteByteGo Newsletter

kept by eddie jun 16

LLM inference splits into compute-bound prefill and memory-bound decode, driving optimization techniques like batching, quantization, speculative decoding, and disaggregation.

A CPU that runs entirely on GPU
nCPU implements a complete CPU entirely with neural networks on a GPU, achieving 100% accuracy on integer arithmetic with multiplication 12x faster than addition.
also on #gpu-computing, #software-engineering, #developer-tools
Yes, AGI Can Happen - A Computational Perspective
Contra Tim Dettmers, Dan Fu argues current AI systems significantly underutilize hardware, models lag hardware buildout, and multiple paths exist to useful AGI.
also on DeepSeek-V3, #software-engineering
The Architecture Behind Open-Source LLMs
Open-weight LLMs now uniformly use Mixture-of-Experts transformers, with key differences in attention mechanisms, expert count, post-training via RL, and permissible licenses.
also on DeepSeek-V3, #developer-tools
Greetings, Earthlings: Philip Johnston of Starcloud on Data Centers in Space
Falling launch costs and rising terrestrial constraints will make space-based AI data centers cheaper than Earth-based ones within a decade, potentially creating a trillion-dollar annual CapEx market for inference workloads.
also on #ai-inference, Nvidia
The emerging role of SRAM-centric chips in AI inference
SRAM-centric chips outperform GPUs on memory-bound AI inference workloads, particularly decode, by placing memory near compute for faster bandwidth.
also on #ai-inference, Nvidia