700 TPS on Kimi K3: A Case for TPU Megakernels

kept by eddie sep 25

Inferact's TPU‑v7 megakernel for Kimi K3 achieves over 700 tokens per second with speculative decoding, nearly double the GB200 baseline at batch size 1.

China’s free Kimi K3 AI model shakes up global tech market - Rest of World
China's free Kimi K3 AI model lets governments run top-tier AI locally, avoiding costly US cloud rentals and enabling sovereign data control.
also on Kimi K3, #cloud-computing, #ai
Kimi K3 Architecture Overview and Notes
The article outlines the Kimi K3 architecture, detailing its modular design, training pipeline, and inference optimizations for large‑scale language models.
also on Kimi K3, #ai
Mac mini alternatives for local LLMs: M6, M5 and Strix Halo
For local LLMs, memory bandwidth and size determine token speed and model capacity; Strix Halo offers more memory per dollar, while Apple Macs with higher bandwidth generate tokens faster.
also on #benchmarks, #performance, #hardware, #ai