Why didn't we get GPT-2 in 2005?

kept by starwatcher oct 02

GPT-2 wasn't created in 2005 because insufficient compute investment and unclear payoff prevented the feedback loop that later drove rapid LLM advances.

Pretraining progress is mostly coming from data
From 2019 to 2025, data improvements contributed 3.24× more compute‑efficiency gains than model changes, delivering 12× versus 3.7× efficiency gains at 1e19 FLOPs.
also on GPT-2, #machine-learning, #ai
ML promises to be profoundly weird
The article argues that while ML models are improving rapidly, they remain fundamentally flawed 'idiots' whose capabilities and limitations are poorly understood.
also on GPT-2, #machine-learning, #ai
Don't let the LLM speak, just probe it. - by James Padolsey
By extracting an LLM's hidden state at the last prompt token and training a small MLP, you get a fast, calibrated classifier for any English criterion.
also on IBM, #machine-learning, #ai
DSHR's Blog: Small Is Beautiful
The article argues that AI firms must generate $2 trillion by 2030 to justify their massive capex, but rising costs, cheaper Chinese open‑weight models, and efficient local inference threaten that goal.
also on IBM, #ai