10x Data Efficiency - NanoGPT Slowrun - Q

kept by eddie mar 20

NanoGPT Slowrun achieves 10x data efficiency via ensembles, chain distillation, heavy regularization, and looping, enabling scale with compute instead of data.

How To Scale Your Model
This online book teaches LLM scaling through roofline analysis, TPU/GPU hardware, Transformer math, and parallelism techniques to estimate training and inference costs.
also on #scaling-laws, #machine-learning, #ai-research
Engram: How DeepSeek Added a Second Brain to Their LLM | rewire.it
DeepSeek's Engram architecture adds conditional memory via N-gram lookup tables to LLMs, improving knowledge and reasoning benchmarks by offloading static pattern reconstruction from neural computation.
also on #scaling-laws, #machine-learning, #ai-research