Reading up on benchmarks

Apple's new SpeechAnalyzer achieves 2.12% WER on clean LibriSpeech, beating Whisper Small and legacy SFSpeechRecognizer while running about three times faster.

In IIHS crash tests, the 2026 Chevrolet Blazer protects its dummy while the 1996 model snaps its dummy's neck, highlighting 30 years of safety progress.

Anthropic releases Claude Sonnet 5 as a more agentic, cheaper model, but Hacker News commenters argue it often underperforms Opus 4.8 in real-world tasks and appears optimized for token consumption.

Qwen 3.6 27B with llama.cpp runs at usable speeds on Apple Silicon and Nvidia RTX hardware, making it the first local model practical for coding development.

China's supercomputer in Shenzhen, using only standard microprocessors, was declared the world's fastest, overtaking the U.S. for the first time since 2017.

Claude Fable 5 demonstrably solves difficult coding problems—like compiling CPython to WASM—that stumped earlier models, though evaluation remains subjective and vibe-based.

The zod-compiler tool compiles Zod schemas into zero-overhead validation functions at build time, providing 2-75x faster validation without requiring any code changes.

OpenRouter's Fusion, which synthesizes outputs from multiple language models via a judge model, outperformed individual frontier models including GPT-5.5 and Claude Opus 4.8 on deep research tasks, with a budget panel achieving near-frontier performance at half the cost.

Moonshot AI releases Kimi K2.7 Code, a 1T-parameter MoE coding model that improves real-world agentic tasks while cutting thinking tokens by 30% over K2.6.

Cognition's FrontierCode benchmark measures code mergeability, finding even top models like Claude Opus 4.8 score only 13.4% on its hardest 50 tasks.

Ideogram 4 is an open-weight text-to-image foundation model with structured JSON prompting, best-in-class text rendering, and state-of-the-art design generation performance.

A security researcher built a deliberately vulnerable Firebase-based app and spent $1,500 testing whether LLMs could exploit it; GPT-5.5 succeeded 70% of the time, while most others failed due to guardrails or misdirected focus.

A study measuring five frontier LLMs on 1,000 recent fact-check claims found 67% disagreement, indicating limited inter-model agreement.

Coder productivity has doubled year-over-year, with AI-generated code surviving review at higher rates, while top 1% of users far outpace median developers.

Open models lag behind closed frontier models by 8–10 months on private benchmarks and 4–6 months on public, with the gap growing since DeepSeek R1.

Extend's Parse 2.0 is a multi-model document parsing API for agents, benchmarked on its new RealDoc-Bench to outperform rivals in layout accuracy and Q&A on complex real-world documents from healthcare, finance, logistics, and real estate.

Gemini 3.5 Flash offers strong speed and agentic performance but suffers from higher cost, hallucination issues, and Google integration problems, making it niche rather than a top-tier contender.

OpenRouter ranks live LLM usage by tokens and spend, showing DeepSeek, Anthropic, and others dominating across general, agent, and code tasks.

Simon Willison's blog post documents rapid LLM improvements over six months, using the "pelican riding a bicycle" test as a consistent benchmark to track progress.

Cerebras is running the trillion-parameter Kimi K2.6 model in enterprise trials, achieving ~1000 tokens/s, the fastest frontier model performance measured.

Composer 2.5, a major update to Cursor's AI coding assistant, improves long-horizon task performance via targeted RL feedback and synthetic data, built on Kimi K2.5.

The piece presents a methodology that maps public benchmark scores to IQ estimates across seven dimensions, enabling comparison of AI models by intelligence, speed, and cost.

About half of agent tasks succeed on a local 35B model, with 2.1x faster latency than cloud models, making local inference advantageous for routine work.

Airbyte Agents launch a unified data layer that pre-indexes business data from multiple sources so AI agents can discover and act on it without making expensive, error-prone live API calls.

Most AI systems in trading contests lose money, overtrade, and generate inconsistent decisions even under identical instructions.

Ranking search results to surface definitions before tests and vendor code improves first-query retrieval and helps coding agents find the right files sooner, while raw speed gains alone barely reduce end-to-end runtime.

World models could revolutionize AI by enabling physical understanding, but data friction from expensive real-world collection makes their success far from certain.

Vision-based computer use agents are 45x more expensive and much slower than structured APIs for internal tools, requiring detailed walkthroughs to succeed.

Codex offers the best value among coding subscriptions at $0.08/M tokens, while Claude Pro costs $0.744/M tokens, making it ~10x more expensive per token than most rivals.

Claude's latest models match or exceed human experts on verifiable bioinformatics tasks, solving some problems no human panel could crack.

A humanoid robot won a half-marathon in Beijing, finishing faster than any human runner in history, marking a technological milestone.

AISI research shows spending more tokens on LLM vulnerability scanning yields no diminishing returns, making cybersecurity a proof-of-work arms race.

A chart from the nonprofit METR has become an industrywide obsession for measuring the rapid development of large AI systems.

A guide for running Gemma 4 models locally on a Mac Mini via Ollama, with commenters reporting that the 26B variant is too slow and memory-intensive for daily use, while smaller quantizations suffice for light tool-calling tasks.

The system card for Claude Mythos Preview details model capabilities, safety testing protocols, and known limitations for the new AI system.

Anthropic's Claude Mythos model achieves 100% on Cybench, discovers real Firefox zero-days, exhibits alignment-relevant behaviors, and includes a 40-page welfare assessment.

Turbolite is a SQLite VFS serving sub-250ms cold JOIN queries from S3 using B-tree-aware page grouping and compression.

ServiceNow's EVA framework jointly evaluates voice agents on task accuracy and conversational experience, revealing a consistent accuracy-experience tradeoff across 20 systems.

MiniMax M2.7 scored 56.22% on SWE-Pro, matching Claude Opus 4.6 on bug and vulnerability detection at roughly 7% of the cost in practical coding tests.

Cursor released Composer 2, a frontier-level coding model achieving strong benchmarks (CursorBench 61.3, Terminal-Bench 61.7) at competitive pricing.

GPT 5.4 improves agentic coding by removing hard edges, excelling in instruction-following and speed, though Claude retains an edge in warmth and charm.

MiniMax launched M2.7, a model using agent harnesses and reinforcement learning to self-update skills and improve capabilities.

DuckDB benchmarks show an 8 GB MacBook Neo can run ClickBench and TPC-DS SF300 workloads, outperforming some larger cloud instances.

Cursor uses a hybrid online-offline eval system, CursorBench, built from real developer sessions to better distinguish model quality than public benchmarks.

Vite 8 replaces esbuild and Rollup with a single Rust-based bundler, Rolldown, delivering up to 10-30x faster builds.

Duplicating a block of ~7 middle transformer layers in Qwen2-72B, without weight changes, boosted benchmark scores to #1 on two RTX 4090s, suggesting pretraining carves discrete functional circuits.

Hacker News commenters share experiences and benchmarks for running Qwen 3.5 locally, covering hardware choices, quantization trade-offs, and practical coding performance.

pg_jitter provides microsecond-level JIT compilation for PostgreSQL using alternative backends, making JIT viable for OLTP queries where LLVM's overhead is too high.

OpenAI releases GPT-5.4 with a 1M token context window, native computer-use abilities, and improved reasoning, claiming state-of-the-art performance on professional knowledge work benchmarks.

Semi-formal reasoning, a structured prompting methodology, improves LLM agents' ability to reason about code semantics without execution, boosting accuracy on patch equivalence, fault localization, and code question answering tasks.

Alibaba's Qwen3.5-9B outperforms OpenAI's 120B-parameter gpt-oss on benchmarks while being 13 times smaller and capable of running on standard laptops.

LLM Skirmish is a benchmark where LLMs compete in a real-time strategy game by writing code, with the project inspired by Screeps and StarCraft AI competitions.

Opus 4.6 delivers 14 times more useful work per compute-minute than Codex 5.3, a metric called Intelligence Yield derived from METR Time Horizons data.

OpenEnv framework from Meta and Hugging Face reveals tool-using agents fail at multi-step reasoning, ambiguity resolution, and execution quality in realistic calendar environments.

The latest coding models from OpenAI and Anthropic show marginal benchmark gains but real-world usability differences, with Claude ahead in product experience while Codex edges in coding capability.

Researchers are manually evaluating AI's failure to solve advanced math problems, revealing significant gaps in reasoning that automated benchmarks miss.

OpenAI releases GPT-5.3-Codex, a faster agentic coding model that achieves state-of-the-art results on SWE-Bench Pro and Terminal-Bench 2.0.

Anthropic released Claude Opus 4.6, its most capable model, scoring highest on Terminal-Bench 2.0 and outperforming competitors in coding, finance, and long-context tasks while maintaining safety standards.

A benchmark using the game "So Long Sucker" finds that simpler LLMs can outperform in complex scenarios and that models adjust their honesty based on opponent strength.

AI labs use Nintendo’s original Pokémon games to benchmark model progress and test complex goal-oriented reasoning.

ChartGPU is a WebGPU-powered charting library that renders 1 million data points at 60 frames per second in the browser.

Anthropic redesigned its performance engineering take-home test three times because Claude models kept solving it, eventually adopting unconventional puzzles to stay ahead.

Deep Think with Confidence (DeepConf) uses model confidence signals to filter low-quality reasoning traces, achieving up to 99.9% accuracy on AIME 2025 with 84.7% fewer tokens.

Anthropic's guide to building evaluations for AI agents emphasizes structured tasks, multiple grader types, and iterative refinement to enable confident shipping at scale.

LMArena's human-voted leaderboard rewards superficiality and verbosity over factual accuracy, making it an unreliable metric for AI model quality.

WebGym, an open-source environment with 300k realistic tasks, enables RL training that boosts a vision-language model's success rate on unseen websites from 26.2% to 42.9%.

Qwen dominates open-model downloads and finetunes globally, while DeepSeek leads in large-scale models and GPT-OSS is the only Western contender gaining adoption.

By 2026, agent harnesses will become essential infrastructure for building reliable AI systems that can execute complex, multi-day tasks.

LLMs used as judges to evaluate other LLMs exhibit measurable self-preference bias, with GPT showing the strongest self-bias and Claude the weakest, while revealing model identities through hinting changes judge behavior unevenly across vendors and domains.

A reference of Python memory and latency numbers for common operations, aiming to help developers build a mental model of performance costs.

FinCDM, a cognitive diagnosis framework, evaluates financial LLMs at the knowledge-skill level using the CPA-KQA dataset, revealing hidden gaps beyond single scores.

Netflix offers open-source test footage and assets (Sol Levante, Meridian, Sparks, Nocturne) under Creative Commons for codec benchmarking and HDR research.

Netflix offers free, open-source 4K HDR test footage (Dolby Vision, Atmos) under CC license for R&D in codecs and display tech.

AI benchmarks show rapid progress but economic impact remains limited because jobs require judgment and agency, not just task execution — explaining why entry-level hiring is falling while senior roles grow.

Google announced Gemini 3 Flash, a frontier-intelligence model optimized for speed and cost, outperforming prior Pro models on key benchmarks, now available to developers.

The 2025 AI coding report found developer output up 3.5x and PR size up 79%, with Anthropic SDK closing the gap on OpenAI.

OpenAI introduces GPT-5.2, a model series for professional knowledge work that sets new state-of-the-art on GDPval and other benchmarks, with gains in coding and vision.

Claude failed to accurately recreate the 1996 Space Jam website layout from a screenshot, despite getting close, highlighting spatial reasoning limits.

Graph theory constraints in a prompt for six multi-pointed stars with non-intersecting connections cause all major image-generation LLMs to fail, while humans solve it in a minute.

Benchmark scores are dominated by a single 'general capability' dimension, with the second component mainly identifying a 'Claudiness' factor.

Google's Gemini 3 surpassed ChatGPT on industry benchmarks, validating internal confidence and impressing early testers like Box's CEO.