How To Scale Your Model

kept by eddie may 06

This online book teaches LLM scaling through roofline analysis, TPU/GPU hardware, Transformer math, and parallelism techniques to estimate training and inference costs.

Why AGI Will Not Happen — Tim Dettmers
Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.
also on Transformer, #scaling-laws, #ai-research
How did ‘large’ language models get that way? The role of Transformers and Pretraining in GPT - LessWrong 2.0 viewer
The transformer architecture and pretraining enabled enormous scaling of language models, making 'large language models' a fitting description.
also on Transformer, #machine-learning, #ai-research, #software-engineering
Designing AI resistant technical evaluations
Anthropic redesigned its performance engineering take-home test three times because Claude models kept solving it, eventually adopting unconventional puzzles to stay ahead.
also on TPU, #software-engineering, #developer-tools
10x Data Efficiency - NanoGPT Slowrun - Q
NanoGPT Slowrun achieves 10x data efficiency via ensembles, chain distillation, heavy regularization, and looping, enabling scale with compute instead of data.
also on #scaling-laws, #machine-learning, #ai-research
Engram: How DeepSeek Added a Second Brain to Their LLM | rewire.it
DeepSeek's Engram architecture adds conditional memory via N-gram lookup tables to LLMs, improving knowledge and reasoning benchmarks by offloading static pattern reconstruction from neural computation.
also on #scaling-laws, #machine-learning, #ai-research