Presentation
An Energy-Efficient Dataflow Architecture for Efficient MoE Model Inference
DescriptionThe Mixture-of-Experts (MoE) model sparsely activates a subset of experts for each token, causing each expert to process a varying number of tokens during inference. However, existing GPU-based MoE inference frameworks adopt a fixed tensor size for the inputs to each expert, requiring padding when an expert receives fewer tokens, which leads to substantial redundant computation.
In response, we propose a token-stationary dataflow that uniformly abstracts the multiplication between an expert's parameters and input tensors with varying token counts. Based on this dataflow, we design a reconfigurable systolic array that eliminates padding-incurred redundant computation. Evaluation demonstrates that our design outperforms the state-of-the-art GPU-based MoE inference frameworks significantly.
In response, we propose a token-stationary dataflow that uniformly abstracts the multiplication between an expert's parameters and input tensors with varying token counts. Based on this dataflow, we design a reconfigurable systolic array that eliminates padding-incurred redundant computation. Evaluation demonstrates that our design outperforms the state-of-the-art GPU-based MoE inference frameworks significantly.
Event Type
Research Manuscript
TimeTuesday, July 2810:54am - 11:06am PDT
LocationMtg Room 101A
Similar Presentations