Presentation
SkiST: Memory-Efficient Fine-Tuning of Spiking Neural Networks via Spatio-Temporal Adaptation
DescriptionBuilding high-performance Spiking Neural Networks (SNNs) involves a trade-off among competing paradigms. While unsupervised rules like STDP offer efficiency and ANN-to-SNN conversion provides a simple path, direct training with Backpropagation Through Time (BPTT) is the most effective route for achieving high accuracy. However, BPTT's effectiveness comes at a steep price: its memory cost scales linearly with timesteps, creating a resource barrier that limits the information capacity and practicality of SNNs. This makes \emph{memory-efficient} training methods critical for real-world deployment.
To this end, we propose the \underline{Ski}pping-\underline{S}ide-\underline{T}uning (SkiST) framework, which leverages both spatial and temporal redundancy in SNNs for \emph{memory-efficient} fine-tuning. On the spatial side, SkiST introduces side networks and applies low-rank approximation with adaptive rank filtering to compress trainable parameters. On the temporal side, it employs Dynamic Sparse BPTT, selectively skipping non-critical timesteps during gradient propagation while compensating for information loss with exponentially decayed gradients. Experiments show SkiST reduces GPU memory by up to 50\% over full fine-tuning while maintaining competitive accuracy. On an SNN-based language model, SkiST requires 23.2\,GB of memory for 1024-token sequences, compared to 40.3\,GB for the baseline, with minimal accuracy degradation. This work enables the deployment of adaptable, efficient spiking models on resource-constrained devices.
To this end, we propose the \underline{Ski}pping-\underline{S}ide-\underline{T}uning (SkiST) framework, which leverages both spatial and temporal redundancy in SNNs for \emph{memory-efficient} fine-tuning. On the spatial side, SkiST introduces side networks and applies low-rank approximation with adaptive rank filtering to compress trainable parameters. On the temporal side, it employs Dynamic Sparse BPTT, selectively skipping non-critical timesteps during gradient propagation while compensating for information loss with exponentially decayed gradients. Experiments show SkiST reduces GPU memory by up to 50\% over full fine-tuning while maintaining competitive accuracy. On an SNN-based language model, SkiST requires 23.2\,GB of memory for 1024-token sequences, compared to 40.3\,GB for the baseline, with minimal accuracy degradation. This work enables the deployment of adaptable, efficient spiking models on resource-constrained devices.
Event Type
Research Manuscript
TimeMonday, July 272:21pm - 2:34pm PDT
LocationMtg Room 203AB