LLMs are complicated now – Ian’s Blog

kept by eddie jun 22

Modern LLMs have grown complex with many attention variants and mixture-of-experts, echoing the messy evolution of recommendation systems.

Train Your Own LLM from Scratch
A hands-on workshop guides readers through building a ~10M-parameter GPT model from scratch in PyTorch, trainable on a laptop in under an hour.
also on PyTorch, #ai-research, #open-source, #developer-tools
The Architecture Behind Open-Source LLMs
Open-weight LLMs now uniformly use Mixture-of-Experts transformers, with key differences in attention mechanisms, expert count, post-training via RL, and permissible licenses.
also on #llm-architecture, #ai-research, #open-source, #developer-tools
The AI Future Is for Everyone - WSJ
Mark Zuckerberg argues that superintelligence should be democratized so everyone can benefit, warning that centralized control would stifle human potential and innovation.
also on LLaMA, Meta, Anthropic, #open-source, #developer-tools
400 Bad Request
AGI capable of most cognitive work could arrive by 2028–2034, but deployment will lag capability due to verification bottlenecks, uneven automation, and institutional friction.
also on LLaMA, #ai-research, #software-engineering
Autoresearch on an old research idea | Blog
Using Karpathy's Autoresearch loop with Claude Code, the author reduced mean rank by 54% on an old research project, primarily through bug fixes and hyperparameter tuning.
also on Andrej Karpathy, #ai-research, #open-source, #developer-tools