A Guide to AI Inference Engineering - ByteByteGo Newsletter
kept by eddie
LLM inference splits into compute-bound prefill and memory-bound decode, driving optimization techniques like batching, quantization, speculative decoding, and disaggregation.