Presentation
CORE: Compact, Optimized, and Resource-Efficient Compute Engine for FP-INT GEMM Acceleration with Binary Coding Quantization
DescriptionRecent advances in large language models (LLMs) have driven exponential growth in parameter counts, amplifying memory footprints and creating acute computational bottlenecks.
Weight-only quantization is a common mitigation because weights dominate storage and incur less accuracy degradation than activation quantization. However, it leaves activations in floating point (FP), forcing FP execution whose area and energy costs far exceed those of integer (INT) compute. To address this limitation, binary coding quantization (BCQ) lowers this cost by replacing many FP multiplies with FP additions, but still requires FP multiplications for scaling factors and suffers from limited representational capacity, causing non-trivial accuracy loss. In this paper, we propose a sum-of-power-of-two scaling factor–based BCQ (SS-BCQ) that approximates FP scaling factors with a small set of power-of-two terms, eliminating FP multiplications (shift–add only) while preserving accuracy. To efficiently realize SS-BCQ in hardware, we introduce CORE, a fully pipelined FP–INT general matrix-matrix multiplication (GEMM) engine. CORE features (i) an adaptive data-mapping module that computes optimized block-wise activation scales to minimize scaling operations, and (ii) an extra processing unit that allocates a small subset of processing elements to high-importance weights to retain accuracy at low cost. Evaluated on the OPT-6.7B model, SS-BCQ reduces perplexity by 3.14 compared with prior BCQ methods. Implemented in CORE, our approach achieves up to 1.41x higher area efficiency (TOPS/mm^2) and 2.18x higher energy efficiency (TOPS/W) than state-of-the-art FP–INT accelerators, enabling a homogeneous low-precision execution path for on-device LLMs.
Weight-only quantization is a common mitigation because weights dominate storage and incur less accuracy degradation than activation quantization. However, it leaves activations in floating point (FP), forcing FP execution whose area and energy costs far exceed those of integer (INT) compute. To address this limitation, binary coding quantization (BCQ) lowers this cost by replacing many FP multiplies with FP additions, but still requires FP multiplications for scaling factors and suffers from limited representational capacity, causing non-trivial accuracy loss. In this paper, we propose a sum-of-power-of-two scaling factor–based BCQ (SS-BCQ) that approximates FP scaling factors with a small set of power-of-two terms, eliminating FP multiplications (shift–add only) while preserving accuracy. To efficiently realize SS-BCQ in hardware, we introduce CORE, a fully pipelined FP–INT general matrix-matrix multiplication (GEMM) engine. CORE features (i) an adaptive data-mapping module that computes optimized block-wise activation scales to minimize scaling operations, and (ii) an extra processing unit that allocates a small subset of processing elements to high-importance weights to retain accuracy at low cost. Evaluated on the OPT-6.7B model, SS-BCQ reduces perplexity by 3.14 compared with prior BCQ methods. Implemented in CORE, our approach achieves up to 1.41x higher area efficiency (TOPS/mm^2) and 2.18x higher energy efficiency (TOPS/W) than state-of-the-art FP–INT accelerators, enabling a homogeneous low-precision execution path for on-device LLMs.
Event Type
Research Manuscript
TimeTuesday, July 2811:30am - 11:42am PDT
LocationMtg Room 101A
Similar Presentations