Presentation
eLLM: Elastic Memory Management Framework for Efficient LLM Serving
DescriptionGPU memory management is critical for efficient Large Language Model (LLM) serving. LLM memory usage primarily comprises weights, activations, and KV caches. While weights are static, activations and KV caches exhibit dynamic and unpredictable behavior, posing significant memory management challenges. Modern LLM serving systems address this through a dual-level approach: activations inherit static tensor abstractions from deep learning frameworks, while KV caches employ specialized page-table virtualization (i.e., PagedAttention). Although this reduces KV cache fragmentation, the fundamental isolation between activation and KV cache management prevents memory sharing across these spaces, leading to suboptimal utilization and 20\% throughput degradation.
To address these limitations, we propose eLLM, an elastic memory management framework. The core components of eLLM include:(1) Virtual Tensor Abstraction: Decouples the virtual address space of tensors from physical GPU memory, creating a unified and flexible memory pool;(2) Elastic Memory Mechanism: Dynamically adjusts memory allocation through runtime memory inflation and deflation, and leverages CPU memory as an extensible buffer;(3) Lightweight Scheduling Strategy: Employs Service-Level Objective (SLO)-aware policies to optimize memory utilization and effectively balance performance trade-offs under stringent SLO constraints.
Comprehensive evaluations demonstrate that eLLM outperforms state-of-the-art systems, achieving up to 2.32$\times$ higher throughput.
To address these limitations, we propose eLLM, an elastic memory management framework. The core components of eLLM include:(1) Virtual Tensor Abstraction: Decouples the virtual address space of tensors from physical GPU memory, creating a unified and flexible memory pool;(2) Elastic Memory Mechanism: Dynamically adjusts memory allocation through runtime memory inflation and deflation, and leverages CPU memory as an extensible buffer;(3) Lightweight Scheduling Strategy: Employs Service-Level Objective (SLO)-aware policies to optimize memory utilization and effectively balance performance trade-offs under stringent SLO constraints.
Comprehensive evaluations demonstrate that eLLM outperforms state-of-the-art systems, achieving up to 2.32$\times$ higher throughput.
Event Type
Research Manuscript
TimeTuesday, July 284:30pm - 4:42pm PDT
LocationMtg Room 101B