Presentation
Zipfloat: An Ultra-Fast and Lossless Floating-Point Compressor for LLM Inference Systems
DescriptionLLM inference stresses GPU memory with large model weights and KV caches, which requires efficient compression techniques.
But existing approaches either sacrifice accuracy (e.g., quantization) or apply serial entropy coding (e.g., DFloat11) that limits throughput. Both constrain the overall inference efficiency.
In this work, we propose ZipFloat, an LLM-oriented floating-point compressor that achieves massively parallel com-/decom-pression while preserving full precision.
We observe that weights and KV-cache values in LLMs exhibit strong statistical redundancies, yet existing floating-point formats mask this redundancy and hinder effective compression.
To address this, ZipFloat employs
(1) Exponent Sparsification, which redefines the binary representation of floating-points to restore compressibility,
and (2) Bit-Matrix Packing, which leverages this restored structure with GPU-native parallelism to deliver extreme throughput.
Evaluations show that ZipFloat delivers up to 700 GB/s com-/decom-pression throughput, outperforming SOTA methods (e.g., DFloat11) by several orders of magnitude while maintaining comparable compression ratios, thereby reducing over 95\% TTFT and improving over 4x inference throughput in LLM systems.
But existing approaches either sacrifice accuracy (e.g., quantization) or apply serial entropy coding (e.g., DFloat11) that limits throughput. Both constrain the overall inference efficiency.
In this work, we propose ZipFloat, an LLM-oriented floating-point compressor that achieves massively parallel com-/decom-pression while preserving full precision.
We observe that weights and KV-cache values in LLMs exhibit strong statistical redundancies, yet existing floating-point formats mask this redundancy and hinder effective compression.
To address this, ZipFloat employs
(1) Exponent Sparsification, which redefines the binary representation of floating-points to restore compressibility,
and (2) Bit-Matrix Packing, which leverages this restored structure with GPU-native parallelism to deliver extreme throughput.
Evaluations show that ZipFloat delivers up to 700 GB/s com-/decom-pression throughput, outperforming SOTA methods (e.g., DFloat11) by several orders of magnitude while maintaining comparable compression ratios, thereby reducing over 95\% TTFT and improving over 4x inference throughput in LLM systems.
Event Type
Research Manuscript
TimeTuesday, July 284:42pm - 4:54pm PDT
LocationMtg Room 101B
Similar Presentations