How did ‘large’ language models get that way? The role of Transformers and Pretraining in GPT - LessWrong 2.0 viewer

kept by eddie may 04

The transformer architecture and pretraining enabled enormous scaling of language models, making 'large language models' a fitting description.

How To Scale Your Model
This online book teaches LLM scaling through roofline analysis, TPU/GPU hardware, Transformer math, and parallelism techniques to estimate training and inference costs.
also on Transformer, #machine-learning, #ai-research, #software-engineering
Why AGI Will Not Happen — Tim Dettmers
Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.
also on Transformer, #ai-research, #ai
Train Your Own LLM from Scratch
A hands-on workshop guides readers through building a ~10M-parameter GPT model from scratch in PyTorch, trainable on a laptop in under an hour.
also on GPT, #machine-learning, #ai-research
LLMs as Judges: Measuring Bias, Hinting Effects, and Tier Preferences
LLMs used as judges to evaluate other LLMs exhibit measurable self-preference bias, with GPT showing the strongest self-bias and Claude the weakest, while revealing model identities through hinting changes judge behavior unevenly across vendors and domains.
also on GPT, #ai-research