Prompt Caching 201
kept by eddie
Prompt caching reduces latency by up to 80% and costs by up to 90% by reusing computed key/value tensors for repeated prompt prefixes in OpenAI's API.
Short for Read Later. Save links from any browser or phone. Click to save, read to clear. The full tour
One line. Many voicesSeek and you shall find
kept by eddie
Prompt caching reduces latency by up to 80% and costs by up to 90% by reusing computed key/value tensors for repeated prompt prefixes in OpenAI's API.