Presentation
Hbspec: A Hybrid-Bonding-Based Heterogeneous Accelerator for Efficient Tree-Structured Speculative Decoding
DescriptionWe propose HBSpec, a hybrid-bonding (HB)-based heterogeneous accelerator for tree-based speculative LLM inference. Unlike existing near-memory processing (NMP)-enabled heterogeneous accelerators that place processing elements (PEs) on DRAM dies with limited computation capability, HBSpec customizes PEs on the incorporated logic die to enhance NMP computation capacity. HBSpec is also equipped with a communication-aware data mapping policy to reduce cross-bank access overhead, an arithmetic intensity-aware scheduler to dynamically assign operators to the most suitable hardware units, and a Branch-KV pool to eliminate memory fragmentation. Experimental results show that HBSpec outperforms NPU-only, NPU+LPDDR, and NPU+HB baselines by 15.56x, 2.36x, and 2.13x, respectively.
Event Type
Research Manuscript
TimeTuesday, July 2811:06am - 11:18am PDT
LocationMtg Room 101A
Similar Presentations