Reading up on Transformer

This online book teaches LLM scaling through roofline analysis, TPU/GPU hardware, Transformer math, and parallelism techniques to estimate training and inference costs.

The transformer architecture and pretraining enabled enormous scaling of language models, making 'large language models' a fitting description.

Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.