Reflections on 2025 - Samuel Albanie

kept by eddie jan 01

AI progress follows compute scaling, making evaluation increasingly difficult but promising radical improvements in infrastructure and economic decision-making.

How to Eval AI Agents — The 2026 Guide
Evaluating AI agents requires floor-raising error analysis, code-aware offline tests, production monitoring, and a tight feedback loop rather than benchmark-maxxing.
also on #evaluation, #software-engineering
2025 letter | Zhengdong
Zhengdong Wang argues that compute scaling—underpinned by Moore's Law and empirical trends—has driven AI progress more reliably than human innovation, despite repeated underestimation.
also on DeepMind, #scaling-laws
9 theses on AI | Sarthak Munshi
AI progress is constrained by long-task reliability, labor reallocation, cost inefficiencies of general APIs, the declining value of raw coding skills, inadequate benchmark testing, the limits of formal verification without strong specs, memory-bound local hardware advantages, the shift from data to environment-driven training, and the rising competitiveness of US open-weight models.
also on METR, Nvidia, #ai
Import AI 455: Automating AI Research
The essay argues there is a 60%+ chance that no-human-involved AI R&D will occur by end of 2028, based on accelerating AI capabilities in coding, science, and engineering tasks.
also on DeepMind, Claude, #software-engineering
When AI builds itself \ Anthropic
Anthropic presents internal data showing AI systems now write 80% of its code and accelerate research, trending toward autonomous recursive self-improvement.
also on METR, Claude, #software-engineering, #ai