Presentation
Faster-MoA: Low-Latency Tree-Structured MoA Serving with Early Exit and Agent-Aware Prefill-Decode Overlap
DescriptionMixture-of-Agents (MoA) is a widely adopted multi-agent paradigm, but existing MoA systems face two major challenges: excessive agent-to-agent connectivity and poor hardware efficiency. To address these two issues, we propose Faster-MoA, a unified algorithm-system co-design for efficient MoA serving. Faster-MoA has three innovations. First, we replace the conventional all-to-all topology with a hierarchical tree structure that introduces structured sparsity in agent connections. Second, we develop a run-time dynamic agent early-exit mechanism that prunes unnecessary agent connections basing on output semantic similarity and answer confidence. Third, we propose an agent-dependency-aware incremental prefilling mechanism that overlaps prefilling and decoding among agents with data dependencies to reduce inference latency. Together, these three innovations enable Faster-MoA to reduce end-to-end serving latency by up to 90% while achieving similar (only ±1% variation) or even higher task accuracy compared with MoA baselines using all-to-all agent connection.
Event Type
Research Manuscript
TimeMonday, July 272:34pm - 2:47pm PDT
LocationMtg Room 101B