Presentation
Greenllm: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving
DescriptionLarge Language Models (LLMs) are rapidly becoming the backbone of modern cloud services, yet their inference costs are dominated by energy consumption on GPUs. Unlike traditional GPU workloads, LLM inference consists of two distinct stages with different characteristics: the prefill phase, which is latency-sensitive and scales quadratically with prompt length, and the decode phase, which progresses token by token with undetermined length. Current GPU power governors (for example, NVIDIA default) overlook this asymmetry, treating both phases uniformly. The result is mismatched voltage/frequency settings, leading to suboptimal voltage/frequency configurations, head-of-line blocking, and excessive energy consumption.
We introduceGreenLLM, a Service-Level Objectives (SLO) aware serving framework that minimizes GPU energy by explicitly separating prefill and decode control. At ingress, requests are routed into length‑based queues so short prompts avoid head‑of‑line blocking, tightening TTFT. For prefill, GreenLLM collects short traces on a GPU node, fits compact latency–power models over SM frequency, and solves a queueing‑aware optimization to pick energy‑minimal clocks per class. During decode, a lightweight dual‑loop controller tracks throughput (tokens-per-second) and adjusts frequency with hysteretic, fine‑grained steps to hold tail TBT within target bounds. Across Alibaba and Azure trace replays, GreenLLM achieves up to 34% reduction in total energy consumption compared to the default DVFS baseline in Alibaba/Azure trace replays, with no loss of throughput and only less than 3.5% SLO violations increase, demonstrating its effectiveness in the efficient LLM service.
We introduceGreenLLM, a Service-Level Objectives (SLO) aware serving framework that minimizes GPU energy by explicitly separating prefill and decode control. At ingress, requests are routed into length‑based queues so short prompts avoid head‑of‑line blocking, tightening TTFT. For prefill, GreenLLM collects short traces on a GPU node, fits compact latency–power models over SM frequency, and solves a queueing‑aware optimization to pick energy‑minimal clocks per class. During decode, a lightweight dual‑loop controller tracks throughput (tokens-per-second) and adjusts frequency with hysteretic, fine‑grained steps to hold tail TBT within target bounds. Across Alibaba and Azure trace replays, GreenLLM achieves up to 34% reduction in total energy consumption compared to the default DVFS baseline in Alibaba/Azure trace replays, with no loss of throughput and only less than 3.5% SLO violations increase, demonstrating its effectiveness in the efficient LLM service.
Event Type
Research Manuscript
TimeMonday, July 271:30pm - 1:42pm PDT
LocationMtg Room 101B