Presentation
Keeping PIM Busy: Eliminating Execution Overheads for Full Throughput
DescriptionLarge language models (LLMs), such as GPT-3 and Llama-2, impose extreme memory-bandwidth demands, creating severe data movement bottlenecks. Processing-in-Memory (PIM) mitigates this by performing computations near memory; however, current PCIe-based PIM systems deliver only a small fraction of their theoretical throughput. On our multi-channel PIM emulation platform, PIM cores are active for only 6.55% of execution time, primarily due to (1) serialized host–device communication that limits channel-level parallelism, (2) insufficient request-generation capability in conventional DMA engines, and (3) per-bank microarchitectures that serialize batch execution.
We address these bottlenecks with a co-designed memory system and microarchitectural solution: Channel-Level Burst (CL) for autonomous per-channel operand generation, PDMA for high-throughput PIM-oriented request scheduling, and an integrated PIM core for enabling true multi-batch parallelism.
Across GEMM microbenchmarks and four LLMs, CL provides 9.10x speedup, PDMA adds 29.6x, and multi-batch execution contributes 102.4x. Together, they deliver a 145.8x improvement over a baseline multi-channel PIM system, enabling PIM to surpass CPU throughput in memory-bound LLM decode kernels and approach the efficiency of GPUs.
We address these bottlenecks with a co-designed memory system and microarchitectural solution: Channel-Level Burst (CL) for autonomous per-channel operand generation, PDMA for high-throughput PIM-oriented request scheduling, and an integrated PIM core for enabling true multi-batch parallelism.
Across GEMM microbenchmarks and four LLMs, CL provides 9.10x speedup, PDMA adds 29.6x, and multi-batch execution contributes 102.4x. Together, they deliver a 145.8x improvement over a baseline multi-channel PIM system, enabling PIM to surpass CPU throughput in memory-bound LLM decode kernels and approach the efficiency of GPUs.
Event Type
Research Manuscript
TimeTuesday, July 2811:23am - 11:36am PDT
LocationMtg Room 203AB