Reading up on llm-evaluation

Majority-vote LLM judge panels overstate agreement when outputs are correlated, so the authors propose a dependence-aware Ising model aggregator that improves accuracy by 9-14% over baselines.

Andrej Karpathy says LLM evaluation must go beyond simple tasks like drawing a pelican on a bike, urging tougher tests such as probing Opus 5’s capabilities.

Claude Fable 5 demonstrably solves difficult coding problems—like compiling CPython to WASM—that stumped earlier models, though evaluation remains subjective and vibe-based.

A study measuring five frontier LLMs on 1,000 recent fact-check claims found 67% disagreement, indicating limited inter-model agreement.

Iterating a prompt 200 times to improve codebase quality with Claude produces exponentially worse code, introducing bloat and bugs.