Presentation
Pegasus: Personalized and Communication-Efficient Mixture-of-Experts System for Distributed Edge LLM Inference
DescriptionLarge Language Models (LLMs) are increasingly pushed to the edge to reduce latency, yet limited memory and compute make full-model deployment infeasible. Mixture-of-Experts (MoE) offers a lightweight alternative, but edge nodes can host only a few experts, creating persistent mismatches between dynamic query semantics and locally available expert models under resource constraints. We present Pegasus, a collaborative MoE inference system for multi-edge deployments that leverages the spatiotemporal correlations inherent in distributed query semantics. Pegasus integrates three innovations: (i) a similarity-based expert deployment mechanism using an efficient heuristic metric to assess the semantic relevance between queries and experts across nodes and over time; (ii) a personalized gating design that selectively fine-tunes a subset of gating parameters on each node to balance expert accuracy and communication latency; (iii) an intra-node online scheduling algorithm with adaptive batching for efficient memory utilization. Extensive performance evaluations corroborate that Pegasus achieves 11.8x lower inference latency than state-of-the-art distributed MoE frameworks, demonstrating high-throughput and communication-efficient edge LLM inference.
Event Type
Research Manuscript
TimeTuesday, July 283:30pm - 3:42pm PDT
LocationMtg Room 101A