Presentation
NVLLM: A 3D NAND-Centric Architecture Enabling Edge on-Device LLM Inference
DescriptionThe rapid growth of LLMs demands high-throughput, memory-capacity–intensive inference on resource-constrained edge devices, where single-batch decoding remains fundamentally memory-bound. Existing out-of-core GPUs and SSD-like accelerators are limited by DRAM-bound weight movement and inefficient storage-access granularity. We present NVLLM, a 3D NAND-centric inference architecture that offloads FFN computation into the Flash array while executing attention on lightweight CMOS logic with external DRAM. Through wafer-to-wafer stacking, NVLLM tightly integrates multi-plane 3D NAND with compute pipelines, ECC units, and buffers, enabling page-level FFN weight access without DRAM traversal. All GEMM/GEMV operations are decomposed into dot-product primitives executed by out-of-order PE lanes, operating directly on raw NAND reads with integrated ECC. Attention weights remain in DRAM, and a KV-cache–aware scheduler maintains throughput as context grows. Evaluated on OPT and LLaMA models up to 30B parameters, NVLLM achieves 16.7× speedup over A800 out-of-core inference and 1.1×–4.7× improvement over SSD-like designs, with only 2.7\% CMOS area overhead.
Event Type
Research Manuscript
TimeTuesday, July 2812:03pm - 12:16pm PDT
LocationMtg Room 203AB
Similar Presentations