Spiking Neural Networks (SNNs) present strong potential for power-efficient edge applications because of their sparsity and event-driven computation. However, direct training of SNNs is particularly challenging for large and customized transformers because of their non-differentiable spike dynamics. To overcome this challenge, Transformer-to-SNN conversion methods have been proposed to leverage existing architectures, but these approaches often suffer from significant performance degradation and high firing rates, which diminish the efficiency benefits of SNNs. In this paper, we propose a conversion approach inspired by the Few Spikes (FS) method and adapt it to the complex deep unfolding sparse transformer (DUST), enabling the conversion of major transformer components. We validate the proposed spiking DUST on the NVIDIA Jetson Orin Nano platform. Experimental results show that the converted model achieves comparable reconstruction quality to the original DUST while reducing power consumption by approximately 25%, demonstrating its potential for power-efficient edge deployment.
Sadaqa, A, De Weerdt, B, Chen, R, Deligiannis, N & da Silva, B 2026, Power-Efficient Spiking Conversion of Deep Unfolded Transformers. in 2026 IEEE International Symposium on Circuits and Systems (ISCAS 2026) . pp. 1-5.
Sadaqa, A., De Weerdt, B., Chen, R., Deligiannis, N., & da Silva, B. (Accepted/In press). Power-Efficient Spiking Conversion of Deep Unfolded Transformers. In 2026 IEEE International Symposium on Circuits and Systems (ISCAS 2026) (pp. 1-5)
@inproceedings{6abc4b43a14b4c408f6c33960113eb69,
title = "Power-Efficient Spiking Conversion of Deep Unfolded Transformers",
abstract = "Spiking Neural Networks (SNNs) present strong potential for power-efficient edge applications because of their sparsity and event-driven computation. However, direct training of SNNs is particularly challenging for large and customized transformers because of their non-differentiable spike dynamics. To overcome this challenge, Transformer-to-SNN conversion methods have been proposed to leverage existing architectures, but these approaches often suffer from significant performance degradation and high firing rates, which diminish the efficiency benefits of SNNs. In this paper, we propose a conversion approach inspired by the Few Spikes (FS) method and adapt it to the complex deep unfolding sparse transformer (DUST), enabling the conversion of major transformer components. We validate the proposed spiking DUST on the NVIDIA Jetson Orin Nano platform. Experimental results show that the converted model achieves comparable reconstruction quality to the original DUST while reducing power consumption by approximately 25%, demonstrating its potential for power-efficient edge deployment.",
author = "Ahmed Sadaqa and {De Weerdt}, Brent and Ruiqi Chen and Nikos Deligiannis and {da Silva}, Bruno",
year = "2026",
month = may,
day = "24",
language = "English",
pages = "1--5",
booktitle = "2026 IEEE International Symposium on Circuits and Systems (ISCAS 2026)",
}