Reading up on evaluation

OpenAI’s 99.9% AGI score for GPT‑6 Astra relied on its custom Provider Adapter harness, while the benchmark’s standard harness yields only 62.7% for the same model.

Evaluating AI agents requires floor-raising error analysis, code-aware offline tests, production monitoring, and a tight feedback loop rather than benchmark-maxxing.

AI progress follows compute scaling, making evaluation increasingly difficult but promising radical improvements in infrastructure and economic decision-making.