A key function of auditory cognition is the association of characteristic sounds with their corresponding semantics over time. Humans attempting to discriminate between fine-grained audio categories, often replay the same discriminative sounds to increase their prediction confidence. We propose an end-to-end attention-based architecture that through selective repetition attends over the most discriminative sounds across the audio sequence. Our model initially uses the full audio sequence and iteratively refines the temporal segments replayed based on slot attention. At each playback, the selected segments are replayed using a smaller hop length which represents higher resolution features within these segments. We show that our method can consistently achieve state-of-the-art performance across three audio-classification benchmarks: AudioSet, VGG-Sound, and EPIC-KITCHENS-100.
Stergiou, A & Damen, D 2023, Play It Back: Iterative Attention For Audio Recognition. in Proceedings - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, IEEE, 2023 International Conference on Acoustics, Speech and Signal Processing, Rhodes, Greece, 4/06/23. https://doi.org/10.1109/ICASSP49357.2023.10096532
Stergiou, A., & Damen, D. (2023). Play It Back: Iterative Attention For Audio Recognition. In Proceedings - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). IEEE. https://doi.org/10.1109/ICASSP49357.2023.10096532
@inproceedings{8527b74ad92549f49f080661cae5ed72,
title = "Play It Back: Iterative Attention For Audio Recognition",
abstract = "A key function of auditory cognition is the association of characteristic sounds with their corresponding semantics over time. Humans attempting to discriminate between fine-grained audio categories, often replay the same discriminative sounds to increase their prediction confidence. We propose an end-to-end attention-based architecture that through selective repetition attends over the most discriminative sounds across the audio sequence. Our model initially uses the full audio sequence and iteratively refines the temporal segments replayed based on slot attention. At each playback, the selected segments are replayed using a smaller hop length which represents higher resolution features within these segments. We show that our method can consistently achieve state-of-the-art performance across three audio-classification benchmarks: AudioSet, VGG-Sound, and EPIC-KITCHENS-100.",
author = "Alexandros Stergiou and Dima Damen",
note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 International Conference on Acoustics, Speech and Signal Processing, ICASSP ; Conference date: 04-06-2023 Through 10-06-2023",
year = "2023",
month = jun,
doi = "10.1109/ICASSP49357.2023.10096532",
language = "English",
isbn = "9781728163284",
series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",
publisher = "IEEE",
booktitle = "Proceedings - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)",
url = "http://2023.ieeeicassp.org",
}