Session

DescriptionRapid large language and diffusion-based model deployment has resulted in major bottlenecks in serving scalable and efficient inference. This session presents architectural and runtime innovations that improve throughput, energy efficiency, and service predictability across modern accelerator platforms. The works span sparsity-aware GPU kernels, metadata-efficient sparse formats, and erasure-coded KV-cache designs to surpass dense execution and reduce load imbalance in distributed serving. Heterogeneous CPU–GPU co-execution and predictive expert caching enable high-throughput Mixture-of-Experts inference, while hardware-aware routing integrates real-time system signals to meet service level objectives. Low-bit adaptive quantization with dedicated microarchitectures further reduces power without sacrificing accuracy. Additional advances accelerate diffusion-based LLMs through phase-aware optimization and speculative decoding and introduce adaptive operator-level tuning frameworks that eliminate redundant compilation overheads for dynamic-shape workloads. Together, these contributions advance scalable, high-efficiency large-model serving through tightly integrated architectural and runtime co-design.