Presentation
Late Breaking Results: Fisher-Guided Selective Error Reconstruction for Quantized LLMs
DescriptionWe propose FOCUS, a hardware-aware PTQ recovery framework for low-bit LLM inference. FOCUS uses Fisher information to select approximately 1.5% structurally critical row-column intersections and optimizes their corrections via ridge regression to match output error, enabling a small set of parameters to compensate global quantization error. Unlike fixed-rank SVD updates that distribute correction capacity uniformly, FOCUS concentrates precision on error-critical locations while remaining lightweight. It supports latency-hidden CPU-GPU co-execution, where sparse compensation is offloaded to the CPU. Experiments show that FOCUS improves perplexity and downstream accuracy under both 2-bit and 4-bit PTQ, achieving up to 29% higher token throughput on the Llama-3.2-3B model.
Event Type
Late Breaking Results
TimeMonday, July 276:24pm - 6:27pm PDT
LocationExhibit Hall