AI IQ — Intelligently Measuring AI Intelligence

kept by eddie may 14

The piece presents a methodology that maps public benchmark scores to IQ estimates across seven dimensions, enabling comparison of AI models by intelligence, speed, and cost.

How far behind are open models? — LessWrong
Open models lag behind closed frontier models by 8–10 months on private benchmarks and 4–6 months on public, with the gap growing since DeepSeek R1.
also on FrontierMath, DeepSeek, #benchmarks
LMArena is a cancer on AI
LMArena's human-voted leaderboard rewards superficiality and verbosity over factual accuracy, making it an unreliable metric for AI model quality.
also on #ai-evaluation, #benchmarks
AI evals are becoming the new compute bottleneck
AI evaluation costs now rival or exceed training costs, with agent benchmarks reaching $40,000 per sweep and training-in-the-loop evals resisting effective compression.
also on #ai-evaluation, #developer-tools
DSHR's Blog: Distilling The Moat
The article argues that AI companies’ claimed moat from expensive model training is shallow because model distillation allows competitors to cheaply replicate capabilities via repeated API queries.
also on DeepSeek, xAI, OpenAI, Anthropic
‘The biggest decision yet’: Jared Kaplan on allowing AI to train itself | Technology
Anthropic's chief scientist Jared Kaplan says humanity must decide by 2027-2030 whether to let AI systems recursively self-improve, a step that could trigger beneficial intelligence explosion or cause humans to lose control.
also on DeepSeek, xAI, OpenAI, Anthropic