Reading up on autoresearch
9 deep · digging since mar 08
- Karpathy's Autoresearch found a 3-year-old bug in our query engine (and improved performance by 11%) - PostHog
PostHog used a Karpathy-style AI agent to find a 3-year-old ClickHouse bug where `toTimeZone()` disabled primary key usage, and the fix cut query time by 37%
- Autoresearch on an old research idea | Blog
Using Karpathy's Autoresearch loop with Claude Code, the author reduced mean rank by 54% on an old research project, primarily through bug fixes and hyperparameter tuning.
- 'The Karpathy Loop': 700 experiments, 2 days, and a glimpse of where AI is heading
Andrej Karpathy's 'autoresearch' AI agent ran 700 experiments in 2 days, discovering 20 optimizations that sped up language model training by 11%.
- Scaling Karpathy's Autoresearch: What Happens When the Agent Gets a GPU Cluster
Giving Karpathy's autoresearch agent access to 16 GPUs enabled 9x faster optimization by running experiments in parallel and enabling factorial grid searches.
- Shopify/liquid: Performance: 53% faster parse+render, 61% fewer allocations
Shopify CEO Tobias Lütke used the autoresearch pattern with a coding agent to achieve 53% faster parse+render and 61% fewer allocations in the Liquid template engine.
Takes
Gumroad’s test suite of 16,000 tests has been flaky for years. This slowed down shipping tremendously. This week, Gianfranco used @karpathy’s autoresearch and @steipete’s OpenClaw to stabilize our test suite overnight. And his code is open source, so you can (have your agent) do it too. (And our code is open source too so you can see every single fix on GitHub.)
@shl
If you still don’t get the hype about autoresearch, read this.
@aakashgupta
How Autoresearch will change Small Language Models adoption
@_philschmid
I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then: - the human iterates on the prompt (.md) - the AI agent iterates on the training code (.py) The goal is to engineer your agents to make the fastest research progress indefinitely and without any of your own involvement. In the image, every dot is a complete LLM training run that lasts exactly 5 minutes. The agent works in an autonomous loop on a git feature branch and accumulates git commits to the training script as it finds better settings (of lower validation loss by the end) of the neural network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc.
@karpathy