Two different tricks for fast LLM inference

kept by eddie feb 16

A speculative analysis of OpenAI's and Anthropic's fast inference strategies, claiming batch-size optimization and Cerebras hardware enable speed at the cost of model quality, though experts challenge key technical assumptions.

The emerging role of SRAM-centric chips in AI inference
SRAM-centric chips outperform GPUs on memory-bound AI inference workloads, particularly decode, by placing memory near compute for faster bandwidth.
also on #ai-inference, Cerebras
Apple Silicon costs more than OpenRouter
Running local LLMs on Apple Silicon costs about 3x more per million tokens than using OpenRouter, and also runs slower.
also on #ai-inference, Anthropic
Two different tricks for fast LLM inference
Anthropic's fast mode uses low-batch-size inference on the full model, while OpenAI's uses a smaller distilled model on Cerebras chips for much higher speed.
also on Cerebras, #hardware, OpenAI, Anthropic
Software engineering at a proprietary trading company: Optiver
Optiver, a proprietary trading firm, has evolved from latency-focused trading to AI-driven models, building custom hardware and full-stack systems while maintaining high engineering ownership and risk-averse speed.
also on Cerebras, OpenAI, Anthropic
For OpenAI and Anthropic, the Competition Is Deeply Personal - The New York Times
A fight over Pentagon contracts reveals how OpenAI and Anthropic's leaders are feuding over the future of the AI industry.
also on Dario Amodei, OpenAI, #startups, Anthropic