Pretraining progress is mostly coming from data

kept by eddie sep 09

From 2019 to 2025, data improvements contributed 3.24× more compute‑efficiency gains than model changes, delivering 12× versus 3.7× efficiency gains at 1e19 FLOPs.

ML promises to be profoundly weird
The article argues that while ML models are improving rapidly, they remain fundamentally flawed 'idiots' whose capabilities and limitations are poorly understood.
also on GPT-2, #machine-learning, #ai
LAION Big Video Dataset
LAION releases a 10‑million‑hour open video dataset (LAION‑BVD) with 80M downloaded videos and synthetic captions for multimodal pre‑training across video, audio, and image modalities.
also on #data-extraction, #machine-learning, #open-source, #ai
Building a Fast Multilingual OCR Model with Synthetic Data
NVIDIA's Nemotron OCR v2 uses 12 million synthetic training images to achieve near-zero error rates across six languages at 34.7 pages per second on an A100.
also on #data-extraction, #machine-learning, #open-source, #ai