Medium

kept by eddie jul 20

Netflix built an LLM serving platform using vLLM and Triton, exposing an OpenAI‑compatible API and enabling scalable constrained decoding for production workloads.

How LLM Inference Works
LLM inference works by tokenizing input, computing embeddings through transformer layers, then generating tokens autoregressively with KV caching and quantization optimizations.
also on vLLM, #software-engineering, #ai, #developer-tools
Why Open-Source Models Haven't Killed the Big Dogs
Open-source models remain costlier to serve reliably due to infrastructure, utilization, and operational overhead, making hosted APIs from OpenAI and Anthropic more economical despite free model weights.
also on vLLM, #software-engineering, #developer-tools
Subscriptions Will Survive in Exactly Two Places
The subscription model survives only for genuine utilities and continuously-fresh-context services; the middle—static SaaS and exhausted catalogs—collapses as income shocks, context saturation, and AI repricing erode its math.
also on Netflix, #software-engineering, #ai
Build Durable AI Agent Systems
This course teaches how to build production-ready AI agent systems with durable execution, secure sandboxes, memory management, and multi-agent orchestration using TypeScript and Node.js.
also on Netflix, #software-engineering, #developer-tools