Presentation
Exquant: Global Expert Ranking–guided Mixed-Precision Quantization for Efficient MoE Inference
DescriptionMixture-of-Experts (MoE) large language models (LLMs) leverage dynamic routing and sparse activation to improve efficiency and scalability, achieving high performance with reduced computational cost. However, their complex architecture and large memory footprint pose significant challenges for deployment, particularly on resource-constrained hardware.
Post-training quantization (PTQ) is a widely used technique to reduce model size and memory usage. Existing PTQ approaches for MoE models are predominantly layer-wise and task-agnostic, optimizing reconstruction error independently within each layer. As a result, they ignore cross-layer differences in expert importance and fail to leverage task-specific signals, causing pivotal experts to be over-compressed, rarely activated experts to be over-provisioned, and overall accuracy to degrade.
To overcome these limitations, we propose ExQuant, a PTQ framework for MoE LLMs that enables global, expert-level mixed-precision quantization. ExQuant first constructs a Globally Comparable Expert Importance Metric by integrating expert routing frequency and post-ablation performance. Based on this metric, it assigns tiered bit-widths to experts and employs a precision-aware load balancing strategy to dynamically schedule computation across processing elements, fully exploiting slack between low- and high-precision workloads. Experiments demonstrate that ExQuant significantly reduces memory footprint, improves inference efficiency, and achieves $2.87-5.93\%$ accuracy improvement over existing MoE quantization methods. These results validate the effectiveness of global, expert-level mixed-precision quantization for efficient and accurate deployment of MoE LLMs.
Post-training quantization (PTQ) is a widely used technique to reduce model size and memory usage. Existing PTQ approaches for MoE models are predominantly layer-wise and task-agnostic, optimizing reconstruction error independently within each layer. As a result, they ignore cross-layer differences in expert importance and fail to leverage task-specific signals, causing pivotal experts to be over-compressed, rarely activated experts to be over-provisioned, and overall accuracy to degrade.
To overcome these limitations, we propose ExQuant, a PTQ framework for MoE LLMs that enables global, expert-level mixed-precision quantization. ExQuant first constructs a Globally Comparable Expert Importance Metric by integrating expert routing frequency and post-ablation performance. Based on this metric, it assigns tiered bit-widths to experts and employs a precision-aware load balancing strategy to dynamically schedule computation across processing elements, fully exploiting slack between low- and high-precision workloads. Experiments demonstrate that ExQuant significantly reduces memory footprint, improves inference efficiency, and achieves $2.87-5.93\%$ accuracy improvement over existing MoE quantization methods. These results validate the effectiveness of global, expert-level mixed-precision quantization for efficient and accurate deployment of MoE LLMs.
Event Type
Research Manuscript
TimeMonday, July 273:30pm - 3:42pm PDT
LocationMtg Room 101B
Similar Presentations