Presentation
HierPAS: Accelerating Video Generation Models with Hierarchical Precision and Adaptive Sparsity
DescriptionDiffusion Transformer (DiT)-based video generation models (VGMs)
achieve state-of-the-art visual quality through global attention mod-
eling but incur heavy computational overhead due to the quadratic
complexity of attention. In high-resolution or long-duration videos,
attention often dominates inference latency. However, much of this
computation is redundant—many attention scores contribute little
to the output and can be safely skipped or approximated. Fully
exploiting this redundancy requires (1) identifying important re-
gions in large attention maps, (2) determining adaptive retention
ratios across heads and blocks, and (3) handling scores of varying
importance efficiently. We present HierPAS, a hardware–software
co-optimized design that accelerates VGMs using hierarchical pre-
cision and adaptive sparsity. HierPAS employs a lightweight eager
attention method to estimate attention patterns and a sampling-
based entropy analysis to derive head-wise retention ratios with
minimal cost. It applies progressively reduced precision to less
critical regions and integrates a configurable top-𝑘 engine with a
unified multi-precision GEMM engine supporting multiple preci-
sions in one datapath. Evaluations show that HierPAS improves
energy efficiency by up to 178×over NVIDIA H20 and 7.7×over
state-of-the-art accelerators, with negligible loss in video quality.
achieve state-of-the-art visual quality through global attention mod-
eling but incur heavy computational overhead due to the quadratic
complexity of attention. In high-resolution or long-duration videos,
attention often dominates inference latency. However, much of this
computation is redundant—many attention scores contribute little
to the output and can be safely skipped or approximated. Fully
exploiting this redundancy requires (1) identifying important re-
gions in large attention maps, (2) determining adaptive retention
ratios across heads and blocks, and (3) handling scores of varying
importance efficiently. We present HierPAS, a hardware–software
co-optimized design that accelerates VGMs using hierarchical pre-
cision and adaptive sparsity. HierPAS employs a lightweight eager
attention method to estimate attention patterns and a sampling-
based entropy analysis to derive head-wise retention ratios with
minimal cost. It applies progressively reduced precision to less
critical regions and integrates a configurable top-𝑘 engine with a
unified multi-precision GEMM engine supporting multiple preci-
sions in one datapath. Evaluations show that HierPAS improves
energy efficiency by up to 178×over NVIDIA H20 and 7.7×over
state-of-the-art accelerators, with negligible loss in video quality.
Event Type
Research Manuscript
TimeWednesday, July 291:55pm - 2:08pm PDT
LocationMtg Room 101B
Similar Presentations