Deep learning methods have been successfully applied to various computer vision tasks. However, existing neural network architectures do not per se incorporate domain knowledge about the addressed problem, thus, understanding what the model has learned is an open research topic. In this paper, we rely on the unfolding of an iterative algorithm for sparse approximation with side information, and design a deep learning architecture for multimodal image super-resolution that incorporates sparse priors and effectively utilizes information from another image modality. We develop two deep models performing reconstruction of a high-resolution image of a target image modality from its low-resolution variant with the aid of a high-resolution image from a second modality. We apply the proposed models to super-resolve near-infrared images using as side information high-resolution RGB images. Experimental results demonstrate the superior performance of the proposed models against state-of-the-art methods including unimodal and multimodal approaches.
Marivani, I, Tsiligianni, E, Cornelis, B & Deligiannis, N 2019, Multimodal Image Super-Resolution via Deep Unfolding with Side Information. in European Signal Processing Conference (EUSIPCO) 2019. European Signal Processing Conference, vol. 2019-September, IEEE, pp. 1-5, 27th European Signal Processing Conference, A Coruña, Spain, 2/09/19. https://doi.org/10.23919/EUSIPCO.2019.8903106
Marivani, I., Tsiligianni, E., Cornelis, B., & Deligiannis, N. (2019). Multimodal Image Super-Resolution via Deep Unfolding with Side Information. In European Signal Processing Conference (EUSIPCO) 2019 (pp. 1-5). (European Signal Processing Conference; Vol. 2019-September). IEEE. https://doi.org/10.23919/EUSIPCO.2019.8903106
@inproceedings{52e14d6bb73a40fc8172c64b3e4b386c,
title = "Multimodal Image Super-Resolution via Deep Unfolding with Side Information",
abstract = "Deep learning methods have been successfully applied to various computer vision tasks. However, existing neural network architectures do not per se incorporate domain knowledge about the addressed problem, thus, understanding what the model has learned is an open research topic. In this paper, we rely on the unfolding of an iterative algorithm for sparse approximation with side information, and design a deep learning architecture for multimodal image super-resolution that incorporates sparse priors and effectively utilizes information from another image modality. We develop two deep models performing reconstruction of a high-resolution image of a target image modality from its low-resolution variant with the aid of a high-resolution image from a second modality. We apply the proposed models to super-resolve near-infrared images using as side information high-resolution RGB images. Experimental results demonstrate the superior performance of the proposed models against state-of-the-art methods including unimodal and multimodal approaches.",
keywords = "Image super-resolution, sparse coding, multimodal deep learning, designing neural networks",
author = "Iman Marivani and Evangelia Tsiligianni and Bruno Cornelis and Nikolaos Deligiannis",
year = "2019",
month = sep,
doi = "10.23919/EUSIPCO.2019.8903106",
language = "English",
series = "European Signal Processing Conference",
publisher = "IEEE",
pages = "1--5",
booktitle = "European Signal Processing Conference (EUSIPCO) 2019",
note = "27th European Signal Processing Conference, EUSIPCO 2019 ; Conference date: 02-09-2019 Through 06-09-2019",
url = "http://eusipco2019.org/",
}