Presentation
Pefoo-L: A General Framework for Preconditioned Forward-Only Optimizer Enabling LLM Fine-Tuning on the Edge
DescriptionFine-tuning Large Language Models (LLMs) on resource-constrained edge devices is a critical but challenging task, primarily due to the prohibitive memory and computational costs of backpropagation. While forward-only optimizers like MeZO mitigate these costs by eliminating the backward pass, they often suffer from slow and unstable convergence. To address this limitation, we introduce PeFoo, which integrates a carefully designed preconditioning strategy into the forward-only paradigm. Furthermore, we propose PeFoo-L to counteract the memory overhead introduced by the preconditioner. This approach constrains preconditioner storage and weight updates to a single layer per iteration, reducing the overall memory footprint and data traffic.
Event Type
Research Manuscript
TimeTuesday, July 284:06pm - 4:18pm PDT
LocationMtg Room 101A