Presentation
Smoe: Elastic MoE-Based Inference with Serverless Computing via Fine-Grained Expert Scaling
DescriptionMixture-of-Experts (MoE) models are attractive for large-scale inference, but sparse activation and skewed, time-varying expert popularity lead to cost-inefficiency. Serverless computing has become an attractive way due to its fine-grained resource elasticity and billing. Even so, achieving cost-efficient and SLO-compliant scaling for each expert with intra- and inter-layer dependencies under concurrent requests remains fraught with challenges. In this paper, we present sMoE, a topology-aware elastic auto-scaler for serverless MoE inference. Our insight is treating the deployment of a MoE inference pipeline as a DAG of experts and non-MoE segments. Building on this, sMoE is designed as a deep reinforcement learning-based solution. Specifically, it encodes expert- and layer-level runtime features with DAGNN by propagating cross-layer semantics. Coupled with a layer-wise pointer network, sMoE captures intra-layer semantics to jointly decide vertical resources, replica counts, and concurrency settings for each of all experts at runtime. Experimental results show that sMoE reduces serving cost by 21.4%–39.2% compared to state-of-the-art solutions, while meeting stringent end-to-end P95 latency SLOs.
Event Type
Research Manuscript
TimeMonday, July 272:21pm - 2:34pm PDT
LocationMtg Room 101B
Similar Presentations
