The success of deep learning in various tasks, including solving inverse problems, has triggered the need for designing deep neural networks that incorporate domain knowledge. In this paper, we design a multimodal deep learning architecture for guided image super-resolution, which refers to the problem of super-resolving a low-resolution image with the aid of a high-resolution image of another modality. The proposed architecture is based on a novel deep learning model, obtained by unfolding a proximal method that solves the problem of convolutional sparse coding with side information. We applied the proposed architecture to super-resolve near-infrared images using RGB images as side information. Experimental results report average PSNR gains of up to 2.85 dB against state-of-the-art multimodal deep learning and sparse coding models.
Marivani, I, Tsiligianni, E, Cornelis, B & Deligiannis, N 2019, Learned Multimodal Convolutional Sparse Coding for Guided Image Super-Resolution. in 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings., 8803313, Proceedings - International Conference on Image Processing, ICIP, vol. 2019-September, IEEE, pp. 2891-2895, IEEE International Conference on Image Processing 2019, Taipei, Taiwan, Province of China, 22/09/19. https://doi.org/10.1109/ICIP.2019.8803313
Marivani, I., Tsiligianni, E., Cornelis, B., & Deligiannis, N. (2019). Learned Multimodal Convolutional Sparse Coding for Guided Image Super-Resolution. In 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings (pp. 2891-2895). Article 8803313 (Proceedings - International Conference on Image Processing, ICIP; Vol. 2019-September). IEEE. https://doi.org/10.1109/ICIP.2019.8803313
@inproceedings{683dee8f240e4765ba5b8101b9dece60,
title = "Learned Multimodal Convolutional Sparse Coding for Guided Image Super-Resolution",
abstract = "The success of deep learning in various tasks, including solving inverse problems, has triggered the need for designing deep neural networks that incorporate domain knowledge. In this paper, we design a multimodal deep learning architecture for guided image super-resolution, which refers to the problem of super-resolving a low-resolution image with the aid of a high-resolution image of another modality. The proposed architecture is based on a novel deep learning model, obtained by unfolding a proximal method that solves the problem of convolutional sparse coding with side information. We applied the proposed architecture to super-resolve near-infrared images using RGB images as side information. Experimental results report average PSNR gains of up to 2.85 dB against state-of-the-art multimodal deep learning and sparse coding models.",
keywords = "Guided image super-resolution, convolutional sparse coding, multimodal deep neural networks",
author = "Iman Marivani and Evangelia Tsiligianni and Bruno Cornelis and Nikolaos Deligiannis",
year = "2019",
month = sep,
doi = "10.1109/ICIP.2019.8803313",
language = "English",
series = "Proceedings - International Conference on Image Processing, ICIP",
publisher = "IEEE",
pages = "2891--2895",
booktitle = "2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings",
note = "IEEE International Conference on Image Processing 2019, ICIP ; Conference date: 22-09-2019 Through 25-09-2019",
}