This paper presents a mosaic convolution-attention network (MCAN) for demosaicing spectral mosaic images captured using multispectral filter array (MSFA) imaging sensors. MSFA-based multispectral imaging systems acquire multispectral information of a scene in a single snap-shot operation. A complete multispectral image is reconstructed by demosaicing an MSFA-based spectral mosaic image. To avoid aliasing and artifacts in demosaicing, we utilize joint spatial-spectral correlation in a raw mosaic image. The proposed MCAN includes a mosaic convolution module (MCM) and a mosaic attention module (MAM). The MCM extracts features via a learning approach with a margin between splitting the periodic spectral mosaic and keeping the underlying spatial information of the raw image. Based on the strategy of position-sensitive weight sharing, MCM assigns the same weight to pixels with the same relative position in an MSFA. The MAM uses a position-sensitive feature aggregation strategy to describe the loading of mosaic patterns within the feature maps, which gradually reduces mosaic distortion through the attention mechanism. The experimental results on synthetic as well as real-world data show that the proposed scheme outperforms state-of-the-art methods in terms of spatial details and spectral fidelity.
Feng, K, Zhao, Y, Chan, JC-W, Kong, S, Zhang, X & Wang, B 2021, 'Mosaic convolution-attention network for demosaicing multispectral filter array images', IEEE Transactions on Computational Imaging, vol. 7, 9507356, pp. 864-878. https://doi.org/10.1109/TCI.2021.3102052
Feng, K., Zhao, Y., Chan, J. C.-W., Kong, S., Zhang, X., & Wang, B. (2021). Mosaic convolution-attention network for demosaicing multispectral filter array images. IEEE Transactions on Computational Imaging, 7, 864-878. Article 9507356. https://doi.org/10.1109/TCI.2021.3102052
@article{8b2ed5cd98df4914857f3fdf0c3325d6,
title = "Mosaic convolution-attention network for demosaicing multispectral filter array images",
abstract = "This paper presents a mosaic convolution-attention network (MCAN) for demosaicing spectral mosaic images captured using multispectral filter array (MSFA) imaging sensors. MSFA-based multispectral imaging systems acquire multispectral information of a scene in a single snap-shot operation. A complete multispectral image is reconstructed by demosaicing an MSFA-based spectral mosaic image. To avoid aliasing and artifacts in demosaicing, we utilize joint spatial-spectral correlation in a raw mosaic image. The proposed MCAN includes a mosaic convolution module (MCM) and a mosaic attention module (MAM). The MCM extracts features via a learning approach with a margin between splitting the periodic spectral mosaic and keeping the underlying spatial information of the raw image. Based on the strategy of position-sensitive weight sharing, MCM assigns the same weight to pixels with the same relative position in an MSFA. The MAM uses a position-sensitive feature aggregation strategy to describe the loading of mosaic patterns within the feature maps, which gradually reduces mosaic distortion through the attention mechanism. The experimental results on synthetic as well as real-world data show that the proposed scheme outperforms state-of-the-art methods in terms of spatial details and spectral fidelity. ",
author = "Kai Feng and Yongqiang Zhao and Chan, {Jonathan Cheung-Wai} and Seong Kong and Xun Zhang and Binglu Wang",
note = "Funding Information: Manuscript received December 15, 2020; revised April 12, 2021, June 8, 2021, and July 7, 2021; accepted July 26, 2021. Date of publication August 4, 2021; date of current version August 18, 2021. This work was supported in part by the Science, Technology and Innovation Commission of Shenzhen Municipality under Grants JCYJ20170815162956949 and JCYJ20180306171146740, in part by the National Natural Science Foundation of China (NSFC) under Grant 61771391, in part by Key R & D plan of Shaanxi Province under Grant 2020ZDLGY07-11, in part by the Natural Science Basic Research Plan in Shaanxi Province of China under Grant 2018JM6056, in part by Korea National Research Foundation under Grant NRF-2016R1D1A1B01008522, and in part by the Yulin smart energy big data application joint Key Laboratory. (Corresponding author: Yongqiang Zhao.) Kai Feng, Yongqiang Zhao, Xun Zhang, and Binglu Wang are with the Research and Development Institute, Northwestern Polytechnical University at Shenzhen, Shenzhen 518057, China (e-mail: 2018100620@mail.nwpu.edu.cn; zhaoyq@nwpu.edu.cn; xunzhang.zx@gmail.com; wbl921129@gmail.com). Publisher Copyright: {\textcopyright} 2015 IEEE.",
year = "2021",
doi = "10.1109/TCI.2021.3102052",
language = "English",
volume = "7",
pages = "864--878",
journal = "IEEE Transactions on Computational Imaging",
issn = "2333-9403",
publisher = "IEEE",
}