Presentation
Inferweave: Efficient LLM Inference on the MT-3000 Processor
DescriptionThe deployment of large language models (LLMs) is increasingly targeting heterogeneous unified-memory architectures (HUMA) for edge and cost-sensitive computing. However, GPU-centric inference systems perform suboptimally on HUMA due to shared DRAM bandwidth contention and inefficient static operator placement. This paper introduces InferWeave, a bandwidth-aware inference framework that co-designs offline planning and runtime scheduling for HUMA. InferWeave's offline planner formulates operator placement and data staging as a unified optimization problem with DRAM bandwidth as a first-class constraint. At runtime, a bandwidth-aware asynchronous pipeline scheduler enforces these plans by regulating memory access rates, enabling non-blocking execution across host and device. Evaluated on LLaMA and GPT models using the MT-3000 processor, InferWeave achieves up to 2.3x higher throughput than FlexGen and llama2.c, demonstrating its effectiveness in enabling efficient LLM inference on integrated architectures.
Event Type
Research Manuscript
TimeMonday, July 271:55pm - 2:08pm PDT
LocationMtg Room 101B
Similar Presentations