Thoughts on Evals – Raindrop Blog

kept by eddie jan 23

Production monitoring and A/B testing offer more reliable evaluation of AI agents than offline evals, which fail to capture real-world performance.

How to Eval AI Agents — The 2026 Guide
Evaluating AI agents requires floor-raising error analysis, code-aware offline tests, production monitoring, and a tight feedback loop rather than benchmark-maxxing.
also on Braintrust, Raindrop, #ai-agents, #software-engineering, #developer-tools
Introducing Agent Self Diagnostics – Raindrop Blog
Raindrop's Self Diagnostics lets AI agents proactively report failures and capability gaps to developers, providing human-readable context and dashboard tracking.
also on Raindrop, #ai-agents, #software-engineering, #developer-tools
Give your agent its own computer
LangSmith Sandboxes give each AI agent its own hardware-isolated microVM with filesystem, shell, and package manager, enabling secure code execution without risking host infrastructure.
also on Langsmith, #ai-agents, #software-engineering, #developer-tools
How to really stop your agents from making the same mistakes (via @garrytan)
LangChain's LangSmith testing platform uses trajectory evals and trace-to-dataset pipelines to prevent AI agents from repeating mistakes, despite its high valuation and funding.
also on Langsmith, #ai-agents, #software-engineering, #developer-tools
Coding sessions in Linear – Changelog
Linear Agent can now write and review code using Claude Code and Codex, enabling end-to-end issue-to-ship workflows from within Linear.
also on Datadog, Sentry, #ai-agents, #developer-tools