Benchmark Scores = General Capability + Claudiness

kept by eddie nov 24, 25

Benchmark scores are dominated by a single 'general capability' dimension, with the second component mainly identifying a 'Claudiness' factor.

Physical Intelligence, a hot robotics startup, says its new robot brain can figure out tasks it was never taught
Physical Intelligence's new π0.7 model can direct robots to perform unfamiliar tasks by combining learned skills, surprising its own researchers.
also on #generalization, #ai-research
How far behind are open models? — LessWrong
Open models lag behind closed frontier models by 8–10 months on private benchmarks and 4–6 months on public, with the gap growing since DeepSeek R1.
also on Epoch AI, #benchmarks, #ai-research, Claude
On Recursive Self-Improvement (Part I) - by Dean W. Ball
Frontier AI labs are automating research to create vast AI workforces that self-improve, radically accelerating capability growth by 2026.
also on Epoch AI, #ai-research, OpenAI
LLMs as Judges: Measuring Bias, Hinting Effects, and Tier Preferences
LLMs used as judges to evaluate other LLMs exhibit measurable self-preference bias, with GPT showing the strongest self-bias and Claude the weakest, while revealing model identities through hinting changes judge behavior unevenly across vendors and domains.
also on #benchmarks, Gemini, #ai-research, Claude
These Mathematicians Are Trying to Educate A.I. - The New York Times
Researchers are manually evaluating AI's failure to solve advanced math problems, revealing significant gaps in reasoning that automated benchmarks miss.
also on #benchmarks, #machine-learning, #ai-research, OpenAI