Presentation
Ultrasketchllm: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators
DescriptionLarge language models (LLMs) require larger GPU memory space these days, necessitating efficient and extreme weight compression methods. Existing compression methods are either theoretically limited by 1 bit per weight or face severe performance degradation and inefficiency. To deploy LLMs in resource-constrained scenarios, we introduce UltraSketchLLM, compressing LLMs with data sketch. It reduces peak GPU memory footprint with a high compression rate up to 0.5 bit per weight. Combined with hardware-friendly implementation, UltraSketchLLM keeps tolerable performance degradation and extremely low latency overhead with 14.9x speedup compared to naive sketch solution.
Event Type
Research Manuscript
TimeTuesday, July 284:30pm - 4:42pm PDT
LocationMtg Room 101A
Similar Presentations