Without Benchmarking LLMs, You're Likely Overpaying 5-10x

kept by eddie jan 21

Benchmarking LLMs on actual task-specific prompts can find cheaper alternatives that match performance, saving 5-10x on API costs.

AI agents find $4.6M in blockchain smart contract exploits
AI agents autonomously exploited smart contracts in simulation, generating $4.6 million in simulated stolen funds, demonstrating a concrete lower bound on economic harm from AI-driven cyberattacks.
also on Claude Opus 4.5, GPT-5
Tasting Notes: Claude Opus 4.5 - FLORA Blog
Claude Opus 4.5's breakthrough lies in reasoning across visual workflows, synthesizing creative intent from reference images and improving generative pipelines.
also on Claude Opus 4.5, #ai-integration, #ai
Inside OpenAI’s in-house data agent
OpenAI built an internal AI data agent using GPT-5, Codex, and memory to reason over massive datasets, delivering insights in minutes.
also on GPT-5, #ai-integration, #developer-tools
Can AI companies become profitable?
Epoch AI's analysis of GPT-5 finds that while it had a 30% gross margin, it failed to recoup R&D costs during its four-month lifecycle, making AI models currently unprofitable.
also on GPT-5, #business-strategy, #ai
Stripe wants to turn your AI costs into a profit center
Stripe previewed a billing feature that lets AI companies track token usage, pass through costs, and automatically apply a profit markup on underlying model fees.
also on OpenRouter, #pricing, #business-strategy, #developer-tools