Reading up on SWE-bench Verified
2 deep · digging since jan 12
- [2602.02361] SWE-Universe: Scale Real-World Verifiable Environments to Millions
SWE-Universe automatically constructs 807,693 real-world software engineering environments from GitHub PRs via iterative self-verification and in-loop hacking detection.
- Demystifying evals for AI agents
Anthropic's guide to building evaluations for AI agents emphasizes structured tasks, multiple grader types, and iterative refinement to enable confident shipping at scale.