Image fusion

The image fusion process is defined as gathering all the important information from multiple images, and their inclusion into fewer images, usually a single one. This single image is more informative and accurate than any single source image, and it consists of all the necessary information. The purpose of image fusion is not only to reduce the amount of data but also to construct images that are more appropriate and understandable for the human and machine perception.^[1]^[2] In computer vision, multisensor image fusion is the process of combining relevant information from two or more images into a single image.^[3] The resulting image will be more informative than any of the input images.^[4]

In remote sensing applications, the increasing availability of space borne sensors gives a motivation for different image fusion algorithms. Several situations in image processing require high spatial and high spectral resolution in a single image. Most of the available equipment is not capable of providing such data convincingly. Image fusion techniques allow the integration of different information sources. The fused image can have complementary spatial and spectral resolution characteristics. However, the standard image fusion techniques can distort the spectral information of the multispectral data while merging.

In satellite imaging, two types of images are available. The panchromatic image acquired by satellites is transmitted with the maximum resolution available and the multispectral data are transmitted with coarser resolution. This will usually be two or four times lower. At the receiver station, the panchromatic image is merged with the multispectral data to convey more information.

Many methods exist to perform image fusion. The very basic one is the high-pass filtering technique. Later techniques are based on Discrete Wavelet Transform, uniform rational filter bank, and Laplacian pyramid.

Motivation

Multi sensor data fusion has become a discipline which demands more general formal solutions to a number of application cases. Several situations in image processing require both high spatial and high spectral information in a single image.^[5] This is important in remote sensing. However, the instruments are not capable of providing such information either by design or because of observational constraints. One possible solution for this is data fusion.

Methods

Image fusion methods can be broadly classified into two groups – spatial domain fusion and transform domain fusion.

The fusion methods such as averaging, Brovey method, principal component analysis (PCA) and IHS based methods fall under spatial domain approaches. Another important spatial domain fusion method is the high-pass filtering based technique. Here the high frequency details are injected into upsampled version of MS images. The disadvantage of spatial domain approaches is that they produce spatial distortion in the fused image. Spectral distortion becomes a negative factor while we go for further processing, such as classification problem. Spatial distortion can be very well handled by frequency-domain approaches on image fusion. The multiresolution analysis has become a very useful tool for analysing remote sensing images. The discrete wavelet transform has become a very useful tool for fusion. Some other fusion methods are also there, such as Laplacian pyramid based, curvelet transform based etc. These methods show a better performance in spatial and spectral quality of the fused image compared to other spatial methods of fusion.

The images used in image fusion should already be registered. Misregistration is a major source of error in image fusion. Some well-known image fusion methods are:

High-pass filtering technique
IHS transform based image fusion
PCA-based image fusion
Wavelet transform image fusion
Pair-wise spatial frequency matching

Comparative analysis of image fusion methods demonstrates that different metrics support different user needs, sensitive to different image fusion methods, and need to be tailored to the application. Categories of image fusion metrics are based on information theory^[4] features, structural similarity, or human perception.^[6]

Multi-focus image fusion

Multi-focus image fusion is used to collect useful and necessary information from input images with different focus depths in order to create an output image that ideally has all information from input images.^[2]^[7] In visual sensor network (VSN), sensors are cameras which record images and video sequences. In many applications of VSN, a camera can’t give a perfect illustration including all details of the scene. This is because of the limited depth of focus exists in the optical lens of cameras.^[8] Therefore, just the object located in the focal length of camera is focused and cleared and the other parts of image are blurred. VSN has an ability to capture images with different depth of focuses in the scene using several cameras. Due to the large amount of data generated by camera compared to other sensors such as pressure and temperature sensors and some limitation such as limited band width, energy consumption and processing time, it is essential to process the local input images to decrease the amount of transmission data. The aforementioned reasons emphasize the necessary of multi-focus images fusion. Multi-focus image fusion is a process which combines the input multi-focus images into a single image including all important information of the input images and it’s more accurate explanation of the scene than every single input image.^[2]

Applications

In remote sensing

Image fusion in remote sensing has several application domains. An important domain is the multi-resolution image fusion (commonly referred to pan-sharpening). In satellite imagery we can have two types of images:

Panchromatic images – An image collected in the broad visual wavelength range but rendered in black and white.
Multispectral images – Images optically acquired in more than one spectral or wavelength interval. Each individual image is usually of the same physical area and scale but of a different spectral band.

The SPOT PAN satellite provides high resolution (10m pixel) panchromatic data. While the LANDSAT TM satellite provides low resolution (30m pixel) multispectral images. Image fusion attempts to merge these images and produce a single high resolution multispectral image.

The standard merging methods of image fusion are based on Red–Green–Blue (RGB) to Intensity–Hue–Saturation (IHS) transformation. The usual steps involved in satellite image fusion are as follows:

Resize the low resolution multispectral images to the same size as the panchromatic image.
Transform the R, G and B bands of the multispectral image into IHS components.
Modify the panchromatic image with respect to the multispectral image. This is usually performed by histogram matching of the panchromatic image with Intensity component of the multispectral images as reference.
Replace the intensity component by the panchromatic image and perform inverse transformation to obtain a high resolution multispectral image.

Pan-sharpening can be done with Photoshop.^[9] Other applications of image fusion in remote sensing are available.^[10]

In medical imaging

Image fusion has become a common term used within medical diagnostics and treatment.^[11] The term is used when multiple images of a patient are registered and overlaid or merged to provide additional information. Fused images may be created from multiple images from the same imaging modality,^[12] or by combining information from multiple modalities,^[13] such as magnetic resonance image (MRI), computed tomography (CT), positron emission tomography (PET), and single-photon emission computed tomography (SPECT). In radiology and radiation oncology, these images serve different purposes. For example, CT images are used more often to ascertain differences in tissue density while MRI images are typically used to diagnose brain tumors.

For accurate diagnosis, radiologists must integrate information from multiple image formats. Fused, anatomically consistent images are especially beneficial in diagnosing and treating cancer. With the advent of these new technologies, radiation oncologists can take full advantage of intensity modulated radiation therapy (IMRT). Being able to overlay diagnostic images into radiation planning images results in more accurate IMRT target tumor volumes.

References

^ Zheng, Yufeng; Blasch, Erik; Liu, Zheng (2018). Multispectral Image Fusion and Colorization. SPIE Press. ISBN 9781510619067.
^ ^a ^b ^c M., Amin-Naji; A., Aghagolzadeh (2018). "Multi-Focus Image Fusion in DCT Domain using Variance and Energy of Laplacian and Correlation Coefficient for Visual Sensor Networks". Journal of AI and Data Mining. 6 (2): 233–250. doi:10.22044/jadm.2017.5169.1624. ISSN 2322-5211.
^ Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "Multi-focus image fusion for visual sensor networks in DCT domain". Computers & Electrical Engineering. 37 (5): 789–797. doi:10.1016/j.compeleceng.2011.04.016. S2CID 38131177.
^ ^a ^b Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features". Computers & Electrical Engineering. 37 (5): 744–756. doi:10.1016/j.compeleceng.2011.07.012. S2CID 7738541.
^ AL Smadi, Ahmad (18 May 2021). "Smart pansharpening approach using kernel-based image filtering". IET Image Processing. 15 (11): 2629–2642. doi:10.1049/ipr2.12251. S2CID 235632628.
^ Liu, Z.; Blasch, E.; Xue, Z.; Langaniere, R.; Wu, W. (2012). "Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence. 34 (1): 94–109. doi:10.1109/tpami.2011.109. PMID 21576753. S2CID 9248856.
^ Naji, M. A.; Aghagolzadeh, A. (November 2015). "Multi-focus image fusion in DCT domain based on correlation coefficient". 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI). pp. 632–639. doi:10.1109/KBEI.2015.7436118. ISBN 978-1-4673-6506-2. S2CID 44524869.
^ Naji, M. A.; Aghagolzadeh, A. (November 2015). "A new multi-focus image fusion technique based on variance in DCT domain". 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI). pp. 478–484. doi:10.1109/KBEI.2015.7436092. ISBN 978-1-4673-6506-2. S2CID 29215692.
^ Pan-sharpening in Photoshop
^ "Beyond Pan-sharpening: Pixel-level Fusion in Remote Sensing Applications" (PDF). Archived from the original (PDF) on 2015-09-01. Retrieved 2013-03-05.
^ James, A.P.; Dasarathy, B V. (2014). "Medical Image Fusion: A survey of state of the art". Information Fusion. 19: 4–19. arXiv:1401.0166. doi:10.1016/j.inffus.2013.12.002. S2CID 15315731.
^ Gooding, M.J.; et al. (2010). "Investigation into the fusion of multiple 4-D fetal echocardiography images to improve image quality". Ultrasound in Medicine and Biology. 36 (6): 957–66. doi:10.1016/j.ultrasmedbio.2010.03.017. PMID 20447758.
^ Maintz, J.B.; Viergever, M.A. (1998). "A survey of medical image registration". Medical Image Analysis. 2 (1): 1–36. CiteSeerX 10.1.1.46.4959. doi:10.1016/s1361-8415(01)80026-8. PMID 10638851.

External links

http://www.math.hcmuns.edu.vn/~ptbao/LVTN/2003/cameras/a161001433035.pdf Archived 2012-03-06 at the Wayback Machine Z. Wang, D. Ziou, C. Armenakis, D. Li, and Q. Li, “A comparative analysis of image fusion methods,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 6, pp. 81–84, Jun. 2005

[Zheng-1] Zheng, Yufeng; Blasch, Erik; Liu, Zheng (2018). Multispectral Image Fusion and Colorization. SPIE Press. ISBN 9781510619067.

[:0-2] M., Amin-Naji; A., Aghagolzadeh (2018). "Multi-Focus Image Fusion in DCT Domain using Variance and Energy of Laplacian and Correlation Coefficient for Visual Sensor Networks". Journal of AI and Data Mining. 6 (2): 233–250. doi:10.22044/jadm.2017.5169.1624. ISSN 2322-5211.

[3] Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "Multi-focus image fusion for visual sensor networks in DCT domain". Computers & Electrical Engineering. 37 (5): 789–797. doi:10.1016/j.compeleceng.2011.04.016. S2CID 38131177.

[fmi-4] Haghighat, M. B. A.; Aghagolzadeh, A.; Seyedarabi, H. (2011). "A non-reference image fusion metric based on mutual information of image features". Computers & Electrical Engineering. 37 (5): 744–756. doi:10.1016/j.compeleceng.2011.07.012. S2CID 7738541.

[5] AL Smadi, Ahmad (18 May 2021). "Smart pansharpening approach using kernel-based image filtering". IET Image Processing. 15 (11): 2629–2642. doi:10.1049/ipr2.12251. S2CID 235632628.

[6] Liu, Z.; Blasch, E.; Xue, Z.; Langaniere, R.; Wu, W. (2012). "Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Survey". IEEE Transactions on Pattern Analysis and Machine Intelligence. 34 (1): 94–109. doi:10.1109/tpami.2011.109. PMID 21576753. S2CID 9248856.

[7] Naji, M. A.; Aghagolzadeh, A. (November 2015). "Multi-focus image fusion in DCT domain based on correlation coefficient". 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI). pp. 632–639. doi:10.1109/KBEI.2015.7436118. ISBN 978-1-4673-6506-2. S2CID 44524869.

[8] Naji, M. A.; Aghagolzadeh, A. (November 2015). "A new multi-focus image fusion technique based on variance in DCT domain". 2015 2nd International Conference on Knowledge-Based Engineering and Innovation (KBEI). pp. 478–484. doi:10.1109/KBEI.2015.7436092. ISBN 978-1-4673-6506-2. S2CID 29215692.

[9] Pan-sharpening in Photoshop

[10] "Beyond Pan-sharpening: Pixel-level Fusion in Remote Sensing Applications" (PDF). Archived from the original (PDF) on 2015-09-01. Retrieved 2013-03-05.

[11] James, A.P.; Dasarathy, B V. (2014). "Medical Image Fusion: A survey of state of the art". Information Fusion. 19: 4–19. arXiv:1401.0166. doi:10.1016/j.inffus.2013.12.002. S2CID 15315731.

[12] Gooding, M.J.; et al. (2010). "Investigation into the fusion of multiple 4-D fetal echocardiography images to improve image quality". Ultrasound in Medicine and Biology. 36 (6): 957–66. doi:10.1016/j.ultrasmedbio.2010.03.017. PMID 20447758.

[13] Maintz, J.B.; Viergever, M.A. (1998). "A survey of medical image registration". Medical Image Analysis. 2 (1): 1–36. CiteSeerX 10.1.1.46.4959. doi:10.1016/s1361-8415(01)80026-8. PMID 10638851.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]