Reading up on ByteByteGo
1 deep · digging since mar 03
- The Architecture Behind Open-Source LLMs
Open-weight LLMs now uniformly use Mixture-of-Experts transformers, with key differences in attention mechanisms, expert count, post-training via RL, and permissible licenses.