Reading up on Transformer

The article visually explains how transformer neural networks work, breaking down attention mechanisms and architecture for intuitive understanding for learners.

This online book teaches LLM scaling through roofline analysis, TPU/GPU hardware, Transformer math, and parallelism techniques to estimate training and inference costs.

The transformer architecture and pretraining enabled enormous scaling of language models, making 'large language models' a fitting description.

Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.