news.ycombinator.com
Speculative Decoding in vLLM on AMD GPUs
Speculative decoding in vLLM enables vLLM to verify multiple draft tokens per target-model pass, boosting throughput on AMD GPUs depending on drafting method and acceptance rate.
One line. Many voicesSeek and you shall find
news.ycombinator.com
Speculative decoding in vLLM enables vLLM to verify multiple draft tokens per target-model pass, boosting throughput on AMD GPUs depending on drafting method and acceptance rate.