Presentation
SAEM: Stage-Aware Expert Management for Memory-Efficient MoE Inference in Chain-of-Thought Reasoning
DescriptionChain-of-thought (CoT) prompting enhances LLM reasoning by decomposing tasks into intermediate steps, but it increases decoding latency and memory usage. Mixture-of-Experts (MoE) models improve scalability via sparse expert activation, yet expert weights often exceed GPU capacity. Existing runtimes treat tokens uniformly, missing structural regularities where consecutive reasoning stages exhibit coherent expert activation patterns. This causes inefficient expert caching and excessive GPU-CPU data movement. We present SAEM, a stage-aware inference runtime that detects reasoning stage boundaries and exploits stage-level coherence in expert activation. SAEM applies stage-aware caching for GPU-CPU placement, combining token packing with in-situ CPU execution. SAEM achieves 1.54× speedup over state-of-the-art baselines under memory constraints.
Event Type
Research Manuscript
TimeTuesday, July 2812:18pm - 12:30pm PDT
LocationMtg Room 101A
Similar Presentations