Why Open-Source Models Haven't Killed the Big Dogs

kept by eddie aug 07

Open-source models remain costlier to serve reliably due to infrastructure, utilization, and operational overhead, making hosted APIs from OpenAI and Anthropic more economical despite free model weights.

Inference Economics 101: Reserved Compute versus Inference APIs
AI inference infrastructure is bifurcating into reserved compute platforms, which monetize predictability, and inference APIs, which monetize utilization through aggregation.
also on BaseTen, Together AI, Fireworks AI, #developer-tools
Yes, AGI Can Happen - A Computational Perspective
Contra Tim Dettmers, Dan Fu argues current AI systems significantly underutilize hardware, models lag hardware buildout, and multiple paths exist to useful AGI.
also on Together AI, #generative-ai, #software-engineering
How prompt caching works - Paged Attention and Automatic Prefix Caching plus practical tips
The article explains how prompt caching works through vLLM's paged attention and automatic prefix caching, detailing KV-cache reuse and practical tips for improving cache hits.
also on SGLang, vLLM, OpenAI, Anthropic, #ai-coding, #software-engineering, #developer-tools
🎙️Inside a Chinese AI Lab: How MiniMax Builds Open Models
MiniMax researcher Olive Song reveals how they debug RL training, handle model hacking, and prioritize alignment and engineering discipline in open-weight model development.
also on SGLang, vLLM, #ai-coding
Cursor admits its new coding model was built on top of Moonshot AI’s Kimi
Cursor admitted its new coding model Composer 2 was built on Moonshot AI's open-source Kimi model, sparking scrutiny over transparency and US-China AI tensions.
also on Fireworks AI, #ai-coding, #developer-tools