Presentation
Reconfigurable 3D-VRRAM LUT-CiM with Greedy G-Shuffle Quantization for End-to-End 4-Bit ViTs in Edge Dense Prediction
DescriptionThe rise of embodied intelligence is driving the deployment of Vision Transformers (ViTs) for dense prediction (DP) tasks on resource-limited edge devices. Look-up-table (LUT)-based digital compute-in-memory (CiM) has emerged as a promising paradigm, delivering high accuracy with superior energy and area efficiency. However, practical ViT deployment on LUT‑CiM for edge DP faces dual hardware and software challenges: (1) LUT cost scales exponentially with bit-width, while NLP‑oriented quantization struggles to push convolution-heavy ViTs below 8 bits, especially in compact on‑chip CiM models; and (2) fixed LUT structures cannot accommodate ViTs' heterogeneous static/dynamic workloads, while unstructured mixed‑precision weights strain memory density and utilization. To overcome these, we propose a software-hardware co-designed low-precision ViT accelerator. Software-wise, we propose Greedy G-shuffle, a lightweight, general quantization method, paired with a PCA‑based, channel‑wise weight assignment, tailored for lightweight ViTs in edge DP tasks. Hardware-wise, at the architectural level, we introduce a reconfigurable LUT-based CiM macro that accelerates both static and dynamic matrix operations with superior energy efficiency, complemented by a high-utilization pipeline- and tensor-parallel dataflow that serves heterogeneous ViT layers. At the circuit level, we employ high-density 1TnR VRRAM with flexible read paths and channel-wise hardware remapping to support mixed-precision INT4/2/1 weights, minimizing area overhead. We demonstrate the first fully 4-bit ViT with INT4 activations and structured INT4/2/1 weights on a reconfigurable 3D 1TnR VRRAM LUT-based CiM macro, achieving a 26.7% improvement in mIoU and a 30.1% reduction in RMSE on the ADE20K and NYUDv2 compared to vanilla round-to-nearest quantization, along with up to 8.9× area efficiency and 9.7× energy efficiency improvements over the state-of-the-art.
Event Type
Research Manuscript
TimeTuesday, July 2811:10am - 11:23am PDT
LocationMtg Room 203AB
Similar Presentations