Presentation
Throughput-Oriented Speculative Decoding via Intra-Device Parallelism
DescriptionLarge Language Model (LLM) serving is crucial for applications like chatbots
and code assistants, yet its implementation remains bottlenecked by low hardware utilization. Speculative decoding improves effective throughput, but still suffers from suboptimal resource partitioning. We introduce SmoothSpec, a speculative decoding system designed to maximize serving throughput by fully leveraging intra-device parallelism. SmoothSpec deploys draft and target models with the same parallelism configurations but allocates asymmetric compute resources at runtime. We design a novel scheduling strategy that dynamically adjusts both the draft parameters for the next step and the resource allocation ratio between the two models, based on the generation quality of draft tokens. We implement SmoothSpec on a single-node LLM serving system and demonstrate its effectiveness across diverse workloads.
and code assistants, yet its implementation remains bottlenecked by low hardware utilization. Speculative decoding improves effective throughput, but still suffers from suboptimal resource partitioning. We introduce SmoothSpec, a speculative decoding system designed to maximize serving throughput by fully leveraging intra-device parallelism. SmoothSpec deploys draft and target models with the same parallelism configurations but allocates asymmetric compute resources at runtime. We design a novel scheduling strategy that dynamically adjusts both the draft parameters for the next step and the resource allocation ratio between the two models, based on the generation quality of draft tokens. We implement SmoothSpec on a single-node LLM serving system and demonstrate its effectiveness across diverse workloads.
Event Type
Research Manuscript
TimeTuesday, July 285:06pm - 5:18pm PDT
LocationMtg Room 101B
Similar Presentations