The fusion of infrared and visible images combines the information from two complementary imaging modalities for various computer vision tasks. Many existing techniques, however, fail to maintain a uniform overall style and keep salient details of individual modalities simultaneously. This paper presents an end-to-end Laplacian Pyramid Fusion Network with hierarchical guidance (HG-LPFN) that takes advantage of pixel-level saliency reservation of Laplacian Pyramid and global optimization capability of deep learning. The proposed scheme generates hierarchical saliency maps through Laplacian Pyramid decomposition and modal difference calculation. In the pyramid fusion mode, all sub-networks are connected in a bottom-up manner. The sub-network for low-frequency fusion focuses on extracting universal features to produce an opposite style while sub-networks for high-frequency fusion determine how much the details of each modality will be retained. Taking the style, details, and background into consideration, we design a set of novel loss functions to supervise both low-frequency images and full-resolution images under the guidance of saliency maps. Experimental results on public datasets demonstrate that the proposed HG-LPFN outperforms the state-of-the-art image fusion techniques.
Yao, J, Zhao, Y, Bu, Y, Kong, S & Chan, JC-W 2023, 'Laplacian Pyramid Fusion Network with Hierarchical Guidance for Infrared and Visible Image Fusion', IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 9, pp. 4630-4644. https://doi.org/10.1109/TCSVT.2023.3245607
Yao, J., Zhao, Y., Bu, Y., Kong, S., & Chan, J. C.-W. (2023). Laplacian Pyramid Fusion Network with Hierarchical Guidance for Infrared and Visible Image Fusion. IEEE Transactions on Circuits and Systems for Video Technology, 33(9), 4630-4644. https://doi.org/10.1109/TCSVT.2023.3245607
@article{c746bbd59cdc435fb9c540a5090dec93,
title = "Laplacian Pyramid Fusion Network with Hierarchical Guidance for Infrared and Visible Image Fusion",
abstract = "The fusion of infrared and visible images combines the information from two complementary imaging modalities for various computer vision tasks. Many existing techniques, however, fail to maintain a uniform overall style and keep salient details of individual modalities simultaneously. This paper presents an end-to-end Laplacian Pyramid Fusion Network with hierarchical guidance (HG-LPFN) that takes advantage of pixel-level saliency reservation of Laplacian Pyramid and global optimization capability of deep learning. The proposed scheme generates hierarchical saliency maps through Laplacian Pyramid decomposition and modal difference calculation. In the pyramid fusion mode, all sub-networks are connected in a bottom-up manner. The sub-network for low-frequency fusion focuses on extracting universal features to produce an opposite style while sub-networks for high-frequency fusion determine how much the details of each modality will be retained. Taking the style, details, and background into consideration, we design a set of novel loss functions to supervise both low-frequency images and full-resolution images under the guidance of saliency maps. Experimental results on public datasets demonstrate that the proposed HG-LPFN outperforms the state-of-the-art image fusion techniques. ",
author = "Jiaxin Yao and Yongqiang Zhao and Yuanyang Bu and Seong Kong and Chan, {Jonathan Cheung-Wai}",
note = "Funding Information: This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 61771391; in part by the Key Research and Development Plan of Shaanxi Province under Grant 2020ZDLGY07-11; in part by the Science, Technology and Innovation Commission of Shenzhen Municipality under Grant JCYJ20170815162956949 and Grant CYJ20180306171146740; and in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Ministry of Science and ICT (MSIT) of South Korea (Development of Artificial Intelligence-Based Video Security Technology and Systems for Public Infrastructure Safety) under Grant 2019-0-00231. Publisher Copyright: {\textcopyright} 1991-2012 IEEE.",
year = "2023",
month = sep,
day = "1",
doi = "10.1109/TCSVT.2023.3245607",
language = "English",
volume = "33",
pages = "4630--4644",
journal = "IEEE Transactions on Circuits and Systems for Video Technology",
issn = "1051-8215",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "9",
}