The Architecture Behind Open-Source LLMs

Open-weight LLMs now uniformly use Mixture-of-Experts transformers, with key differences in attention mechanisms, expert count, post-training via RL, and permissible licenses.