The Architecture Behind Open-Source LLMs

kept by eddie mar 03

Open-weight LLMs now uniformly use Mixture-of-Experts transformers, with key differences in attention mechanisms, expert count, post-training via RL, and permissible licenses.

LLMs are complicated now – Ian’s Blog
Modern LLMs have grown complex with many attention variants and mixture-of-experts, echoing the messy evolution of recommendation systems.
also on #llm-architecture, #ai-research, #open-source, #developer-tools
[2602.21193] On Data Engineering for Scaling LLM Terminal Capabilities
NVIDIA's Nemotron-Terminal models, trained on a new open-source synthetic dataset, dramatically boost terminal task accuracy, with the 32B parameter version matching much larger models on Terminal-Bench 2.0.
also on Qwen3, #open-source
Yes, AGI Can Happen - A Computational Perspective
Contra Tim Dettmers, Dan Fu argues current AI systems significantly underutilize hardware, models lag hardware buildout, and multiple paths exist to useful AGI.
also on DeepSeek-V3, #ai-research
A Guide to AI Inference Engineering - ByteByteGo Newsletter
LLM inference splits into compute-bound prefill and memory-bound decode, driving optimization techniques like batching, quantization, speculative decoding, and disaggregation.
also on DeepSeek-V3, #developer-tools
joke-generator
Training a joke generator on Kimi K2 using rubric-based RL that decomposes humor into verifiable properties like specificity and commitment.
also on Kimi K2, #machine-learning, #ai-research