Abstract
The registration of images from infrared sensors and visible color sensors is a quite difficult problem due to their different phenomena. In this paper, we propose a new method to register visible and infrared images. The proposed approach consists of three main steps. In the first step, SURF (Speeded-Up Robust Features) algorithm is applied for local feature extraction. In the second step, LSS (local self-similarity descriptor) is computed for each extracted feature. Finally, a cross matching process followed by a consistency check in the projective transformation model is performed for feature correspondence and mismatch elimination. Experimental results show the proposed method achieves better accuracy for registering visible and infrared images as compared to state-of-the-art approaches.
Similar content being viewed by others
REFERENCES
X. Liu, Y. Ai, J. Zhang, et al., “A novel affine and contrast invariant descriptor for infrared and visible image registration,” Remote Sens. 10 (4), 658–666 (2018).
B. Zitova and J. Flusser, “Image registration methods: A survey,” Image Vision Comput. 21 (11), 977–1000 (2003).
L. Gottesfeld Brown, “A survey of image registration techniques,” ACM Comput. Surv. 24 (4), 325–376 (1992).
M. Deshmukh and U. Bhosle, “A survey of image registration,” Int. J. Image Process. 5 (3), 245–269 (2011).
J. Flusser and T. Suk, “A moment-based approach to registration of images with affine geometric distortion,” IEEE Trans. Geosci. Remote Sens. 32 (2), 382–387 (1994).
K. Mikolajczyk and C. Schmid, “A performance evaluation of local descriptors,” IEEE Trans. Pattern Anal. Mach. Intell. 27 (10), 1615–1630 (2005).
B. S. Reddy and B. N. Chatterji, “An FFT-based technique for translation, rotation, and scale-invariant image registration,” IEEE Trans. Image Process. 5 (8), 1266–1271 (1996).
D. G. Lowe, “Object recognition from local scale-invariant features,” in Proceeding 7th of International Conference on Computer Vision (ICCV) (1999), pp. 1150–1157.
H. Bay et al., “Speeded-up robust features (SURF),” Comput. Vision Image Understanding 110 (3), 346–359 (2008).
E. Shechtman and M. Irani, “Matching local self-similarities across images and videos,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2007), pp. 18–23.
A. Sedaghat and H. Ebadi, “Distinctive order based self-similarity descriptor for multi-sensor remote sensing image matching,” ISPRS J. Photogramm. Remote Sens. 108, 62–71 (2015).
P. Sourabh and U. C. Pati, “Optical-to-SAR image registration using modified distinctive order based self-similarity operator,” in 2018 IEEE International Students’ Conference on Electrical, Electronics, and Computer Science (SCEECS) (2018).
B. Baheti et al., “A local self-similarity-based vehicle detection approach using single query image,” in 2nd International Conference on Computer Vision and Image Processing (2018), pp. 255–264.
H. M. Chen, P. K. Varshney, and M. K. Arora, “Performance of mutual information similarity measure for registration of multitemporal remote sensing images,” IEEE Trans. Geosci. Remote Sens. 41 (11), 2445–2454 (2003).
Y. S. Heo, K. M. Lee, and S. U. Lee, “Robust stereo matching using adaptive normalized cross-correlation,” IEEE Trans. Pattern Anal. Mach. Intell. 33 (4), 807–822 (2011).
Z. Yi, C. Zhiguo, and X. Yang, “Multi-spectral remote image registration based on sift,” Electron. Lett. 44 (2), 107–108 (2008).
A. Sedaghat, M. Mokhtarzade, and H. Ebadi, “Uniform robust scale-invariant feature matching for optical remote sensing images,” IEEE Trans. Geosci. Remote Sens. 49 (11), 4516–4527 (2011).
J. Li and M. A. Nigel, “A comprehensive review of current local features for computer vision,” Neurocomputing 71 (10–12), 1771–1787 (2008).
X. M. Zhang, X. H. Zhang, F. F. Ji, and Q. F. Yu., “Known landing area rough locating from far distance based on DDM-SIFT,” in IEEE Congress on Image and Signal Processing (2008), pp. 132–135.
J. Dou, Q. Qin, Z. Tu, X. Peng, and Y. Li, “Infrared and visible image registration based on SIFT and sparse representation,” in 28th Chinese Control and Decision Conference (CCDC) (2016), pp. 5420–5424.
X. Mei and H. Ling, “Robust visual tracking using l1 minimization,” in 12th International Conference on Computer Vision (ICCV) (2009), pp. 1–7.
J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” IEEE Trans. Pattern Anal. Mach. Intell. 31 (2), 210–227 (2009).
G. Carneiro and A. D. Jepson, “Flexible spatial configuration of local image features,” IEEE Trans. Pattern Anal. Mach. Intell. 29 (12), 2089–2104 (2007).
J. F. Dou and J. X. Li, “Automatic image mosaic based on SIFT using bidirectional matching,” Adv. Mater. Res. 457–458, 841–847 (2012).
S. Damotharasamy, “Approach to model human appearance based on sparse representation for human tracking in surveillance,” IET Image Process. 14 (11), 2383–2394 (2020).
W. Zhong, H. Lu, and M. H. Yang, “Robust object tracking via sparsity-based collaborative model,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition (2012), pp. 1838–1845.
J. Wang, J. Yang, K. Yu, et al., “Locality-constrained linear coding for image classification,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2010), pp. 3360–3367.
Funding
This work was supported by the by Shanghai University Outstanding Teachers Cultivation Fund Program A30DB1524011-21 and 2015 School Fund Project A01GY15GX48 and Shanghai Second Polytechnic University Mechanical Engineering Key Disciplines XXKZD1603 and the Construction of University Enterprise Cooperation Automobile Electronic Joint Experiment Center, grant no. A11NH182016.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
The authors declare that they have no conflict of interest.
Additional information
Jian-fang Dou. He is currently a teacher with the School of Intelligent Manufacturing and Control Engineering, Shanghai Second Polytechnic University, Shanghai, China. In 2006, obtained the Bachelor Degree, Geography Information System, College of Traffic and Transport, Hebei Polytechnic University, Hebei Province, China. In 2009, obtained the Master Degree, Photogrammetry and Remote Sensing, School of Civil Engineering, Tongji University, Shanghai, China. In 2014, received the PhD Degree from Shanghai Jiaotong University, Shanghai, China. His major interests are object detection and tracking, machine learing, intelligent measurement and control.
Qin Qin. She is currently an Associate Professor with the School of Intelligent Manufacturing and Control Engineering, Shanghai Second Polytechnic University, Shanghai, China. In 2006, received the PhD Degree from the Shanghai Institute of Technical Physics of the Chinese Academy of Sciences, Shanghai, China. Her major interests are intelligent detection, image processing, and pattern recognition.
Zi-mei Tu. She is currently a teacher with the School of Intelligent Manufacturing and Control Engineering, Shanghai Second Polytechnic University, Shanghai, China. In 2012, obtained the Bachelor Degree, Information display and photoelectric technology, Shanghai Second Polytechnic University, Shanghai, China. In 2014, obtained the Master Degree, Shanghai Normal University, China. Her major interests are image segmentation and three-dimension reconstruction.
Rights and permissions
About this article
Cite this article
Dou, J., Qin, Q. & Tu, Z. Multi-Modal Image Registration Based on Local Self-Similarity and Bidirectional Matching. Pattern Recognit. Image Anal. 31, 7–17 (2021). https://doi.org/10.1134/S1054661820040112
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S1054661820040112