Presentation
PHAP: A Pre-Analyzed Head-Wise Attention Pattern for Efficient Sparse Attention in LLM Inference
DescriptionRecent advancements in large language models (LLMs) have demonstrated powerful capabilities across various application domains. At the same time, their quadratic computational complexity with respect to input sequence length remains a major bottleneck for efficient inference and deployment. To mitigate this problem, sparse attention methods utilizing the sparsity of computationally intensive operations in attention of LLM were proposed, but still incur additional overhead to locate important tokens during inference or performance degradation due to utilizing a fixed pattern, neglecting the difference of head-wise sparsity. In this paper, we propose a pre-analyzed head-wise attention pattern (PHAP) sparse attention method, which constructs head-specific patterns by analyzing position-dependent characteristics and the inherent sparsity variations across attention heads. This one-time pre-analyzed pattern construction eliminates runtime overhead during inference. In addition, the proposed method constructs optimal static sparsity patterns tailored to the unique characteristics of each attention head, thereby maintaining the baseline performance. Experimental results show reduced attention computation by more than 75% without additional overhead while maintaining performance comparable to full attention in Llama-3. Moreover, it achieves 1.17x faster inference than conventional sparse attention methods, demonstrating its effectiveness in maximizing the computational efficiency of LLMs.
Event Type
Work in Progress
TimeMonday, July 276:37pm - 6:38pm PDT
LocationExhibit Hall