9 Observations from Building with AI Agents

kept by eddie feb 20

Building AI agents reveals nine lessons including using best models for prototyping, fine-tuning for stable tasks, static typing to reduce hallucinated code, and closed-loop prompt optimization.

We Got Claude to Fine-Tune an Open Source LLM
Hugging Face released an open-source 'skill' that lets Claude Code, Codex, and Gemini CLI autonomously fine-tune LLMs on cloud GPUs and push models to the Hub.
also on #fine-tuning, #developer-tools
[2508.15260] Deep Think with Confidence
Deep Think with Confidence (DeepConf) uses model confidence signals to filter low-quality reasoning traces, achieving up to 99.9% accuracy on AIME 2025 with 84.7% fewer tokens.
also on Qwen 3
The Potential of RLMs
Recursive Language Models (RLMs) mitigate context rot in LLMs by wrapping long contexts in a REPL environment, turning the problem into a coding and reasoning task.
also on DSPy, #ai-agents, #software-engineering, #developer-tools
We Should Take Text Optimization More Seriously
Text optimization—modifying prompts, context, memory, and harnesses—is a legitimate, sample-efficient learning mechanism that deserves the same rigorous study as weight optimization.
also on DSPy, #software-engineering, #developer-tools