We’re introducing a new model benchmark.
And it’s a different kind of benchmark. (Basemark? Vibench?)
A different kind because it’s breathing, constantly updated from millions of builders. Not a closed set of tasks.
For a while now the public benchmark have not been really useful. Many models scoring high on benchmarks with very low real world usability
So we’re introducing to the world a new benchmark that we’re using internally and found extremely useful.
Our benchmark is basically how satisfied millions of users are when using different models.
IMO it’s the closest measurement to how useful a model is in real world use cases.
This metric is also correlated with our own business metrics - conversion, retention, etc.
We called it the frustration meter.
It’s automatically analysing millions of messages daily
It detects bug loops, repeated requests, etc.
We use this to benchmark every model we consider shipping. Not by asking "did it generate correct code." By asking "how did the builder feel after using it."
it’s a good benchmark to measure model degradation. So far in the past few weeks we haven’t found any.
Here's where the top models stand right now, ranked by average frustration score (scale 1 to 5, lower is better):
opus 4.6 - 1.3
sonnet 4.6 - 1.4
opus 4.7 - 1.5
gpt 5.5 - 1.5
gpt 5.4 - 1.6
Gemini 3.1 - 2.2
For app building, Opus 4.6 seems better than 4.7 to a lot of builders. We ran Opus 4.7 50/50 against Opus 4.6 across over 10,000 apps. Frustration riseed by 43%. Turns per request by 19%.
Gemini 3.1 don’t perform well at the moment, I left out of the graph as it made it unclear due to it’s rapid changes in this benchmark.
Quick note - this is all aggregated data, and do not involve reading individual or identifiable conversations.
We’ll keep tracking it and I’ll share it from time to time.
@MaorShlomo
