Reading up on scaling-laws

Dario Amodei argues that AI's exponential progress now demands binding regulation, economic redistribution, and accelerated innovation governance to match the pace of risk.

Epoch AI estimates frontier labs use less than half of global AI compute, but OpenAI and Anthropic may soon dominate, requiring economic transformation to sustain scaling.

Cerebras is running the trillion-parameter Kimi K2.6 model in enterprise trials, achieving ~1000 tokens/s, the fastest frontier model performance measured.

Microservices were originally a technical fix for organizational scaling, but the rise of AI coding agents now demands larger code ownership and new architectural boundaries, creating significant tension.

This online book teaches LLM scaling through roofline analysis, TPU/GPU hardware, Transformer math, and parallelism techniques to estimate training and inference costs.

NanoGPT Slowrun achieves 10x data efficiency via ensembles, chain distillation, heavy regularization, and looping, enabling scale with compute instead of data.

Sunday Robotics raises $165M Series B for a data-first home-robot approach that uses cheap tool-based data collection instead of expensive robot teleoperation.

Designing a universal ID system for all possible objects across all future cosmic eras may require ~800-bit identifiers to avoid collisions.

The combination of large language models with reinforcement learning creates a path for continued improvement, making the argument that AI progress will plateau unlikely.

Frontier AI labs are automating research to create vast AI workforces that self-improve, radically accelerating capability growth by 2026.

Apparent contradictions like OpenAI selling ads don't disprove imminent superintelligence because AI labs must fundraise and hedge against normal-tech scenarios to win the race.

Recent advances in expansion microscopy, protein barcoding, and AI-based connectomics make full human brain emulation plausible within decades, but require massive data acquisition and compute.

DeepSeek's Engram architecture adds conditional memory via N-gram lookup tables to LLMs, improving knowledge and reasoning benchmarks by offloading static pattern reconstruction from neural computation.

Deep Think with Confidence (DeepConf) uses model confidence signals to filter low-quality reasoning traces, achieving up to 99.9% accuracy on AIME 2025 with 84.7% fewer tokens.

Global AI computing capacity from chips has grown 3.3x per year since 2022, doubling every seven months, with NVIDIA supplying over 60%.

Anthropic bets on efficiency and disciplined spending to compete with rivals' massive scale, challenging the dominant AI scaling paradigm.

AI progress follows compute scaling, making evaluation increasingly difficult but promising radical improvements in infrastructure and economic decision-making.

Zhengdong Wang argues that compute scaling—underpinned by Moore's Law and empirical trends—has driven AI progress more reliably than human innovation, despite repeated underestimation.

Gemini 3 Flash likely uses a massive 1.2 trillion-parameter ultra-sparse MoE architecture, activating only 5–30 billion parameters per inference to deliver high intelligence at low cost, though it suffers from token bloat and a 91% hallucination rate on refusals.

OpenAI researcher John Schulman discusses reinforcement learning scaling, LLM usefulness milestones, and lessons for building AI research teams.

Contra Tim Dettmers, Dan Fu argues current AI systems significantly underutilize hardware, models lag hardware buildout, and multiple paths exist to useful AGI.

Tim Dettmers argues AGI will never happen because computation is physical, linear progress requires exponential resources, and GPU improvements have plateaued.

Leading AI researchers argue that current LLM scaling will hit a ceiling, requiring new insights before AGI emerges, pushing timelines back 5-20 years.

The space of cognitive tasks is high-dimensional, so low-dimensional narratives comparing AI and human intelligence are significant oversimplifications.

Andrej Karpathy argues that the space of possible intelligences is vast, with animal intelligence being just one specific point, challenging common intuitions.

Gemini 3's performance leap and Nvidia's $0.5 trillion revenue visibility confirm AI scaling laws are accelerating, not plateauing.

Gemini 3's improvements come from enhancing pre-training and post-training, countering the belief that scaling is no longer effective.

Reinforcement learning provides far fewer bits per FLOP than pretraining until models achieve high pass rates, limiting RLVR's ability to learn new capabilities.