Reading up on llm-architecture
2 deep · digging since mar 03
- LLMs are complicated now – Ian’s Blog
Modern LLMs have grown complex with many attention variants and mixture-of-experts, echoing the messy evolution of recommendation systems.
- The Architecture Behind Open-Source LLMs
Open-weight LLMs now uniformly use Mixture-of-Experts transformers, with key differences in attention mechanisms, expert count, post-training via RL, and permissible licenses.