How LLM Inference Works

kept by eddie nov 24, 25

LLM inference works by tokenizing input, computing embeddings through transformer layers, then generating tokens autoregressively with KV caching and quantization optimizations.

On-Device LLMs: State of the Union, 2026 – Vikas Chandra – AI Research @ Meta
Billion-parameter LLMs now run in real time on phones due to advances in model compression, quantization, and efficient architectures, not just faster chips.
also on AWQ, #machine-learning, #ai, #developer-tools
400 Bad Request
AGI capable of most cognitive work could arrive by 2028–2034, but deployment will lag capability due to verification bottlenecks, uneven automation, and institutional friction.
also on LLaMA, #software-engineering, #ai
LLMs are complicated now – Ian’s Blog
Modern LLMs have grown complex with many attention variants and mixture-of-experts, echoing the messy evolution of recommendation systems.
also on LLaMA, #software-engineering, #developer-tools
Why Open-Source Models Haven't Killed the Big Dogs
Open-source models remain costlier to serve reliably due to infrastructure, utilization, and operational overhead, making hosted APIs from OpenAI and Anthropic more economical despite free model weights.
also on vLLM, Hugging Face, #generative-ai, #software-engineering, #developer-tools
GitHub - kanishkg/endless-terminals
Endless Terminals is an autonomous pipeline that procedurally generates terminal-use tasks without human annotation for training terminal agents with reinforcement learning.
also on vLLM, Hugging Face