Reading up on GPT-2
3 deep · digging since apr 12
- Why didn't we get GPT-2 in 2005?
GPT-2 wasn't created in 2005 because insufficient compute investment and unclear payoff prevented the feedback loop that later drove rapid LLM advances.
- Pretraining progress is mostly coming from data
From 2019 to 2025, data improvements contributed 3.24× more compute‑efficiency gains than model changes, delivering 12× versus 3.7× efficiency gains at 1e19 FLOPs.
- ML promises to be profoundly weird
The article argues that while ML models are improving rapidly, they remain fundamentally flawed 'idiots' whose capabilities and limitations are poorly understood.