Presentation
PD-Swap: Prefill–decode Logic Swapping for End-to-End LLM Inference on Edge FPGAs via Dynamic Partial Reconfiguration
DescriptionAggressively quantized large language models (LLMs), such as BitNet-style 1.58-bit Transformers with ternary weights, make it feasible to deploy generative AI on low-power edge FPGAs. As prompts grow to tens of thousands of tokens, edge hardware performance drops sharply with sequence length due to quadratic prefill cost and rapidly increasing KV-cache bandwidth demands, making context length a first-order systems concern. Recent work with table-lookup-based ternary matrix multiplication on edge FPGAs exposes a fundamental prefill--decode asymmetry: prefill is compute-bound and dominated by dense matrix--matrix operations, whereas decoding is memory-bandwidth-bound and dominated by KV-cache traffic. A static accelerator must provision resources and a single dataflow for both regimes, leading to duplicated attention logic, underutilized fabric, and tight LUT/URAM limits that cap model size and usable context. We propose a prefill--decode disaggregated LLM accelerator that uses Dynamic Patial Reconfiguration (DPR) to time-multiplex the attention module on edge FPGAs. The core table-lookup ternary matrix multiplication and weight-buffering engines remain static, while the attention subsystem is a reconfigurable partition with two phase-specialized dataflows: a compute-heavy, token-parallel prefill engine and a bandwidth-optimized, KV-cache-centric decoding engine. A roofline-inspired model and design space exploration jointly optimize reconfigurable-region size, parallelism under reconfiguration and routability constraints, and partial reconfiguration is overlapped with ongoing computation. Our design achieves state-of-the-art performance, with up to 27~tokens/s decoding throughput. Compared with the prior static design, PD-Swap achieves a 1.3x--2.1x speedup, with larger gains at longer context lengths.
Event Type
Research Manuscript
TimeTuesday, July 284:06pm - 4:18pm PDT
LocationMtg Room 202C
Similar Presentations