Improved covariant local feature detector

Highlights

• The covariant local feature detector is improved by complementary information.

• Keypoints are detected by incorporating the confidence into the predicted positions.

• The proposed method is a general framework fusing two different keypoint detectors.

• State of the art performance has been obtained on four benchmarks.

Abstract

Local feature detection is a fundamental problem in computer vision. Recently, the research of local feature detection has been switched from handcrafted methods to learning based ones, especially deep learning based ones. A recent successful deep learning based feature detector is the covariant local feature detector that conducts keypoint detection by predicting the transformation of keypoints from nearby pixels. Although this method adopts a new detection framework compared to those methods by computing the keypoint’s likelihood, it treats each pixel equally which may incorrectly detect unstable keypoints. On the other hand, other methods computing the keypoint probability could capture different evidence for keypoint detection as well as provide a natural weight for each prediction in the covariant detector. So, fusing information from other detectors into the covariant detector could improve its performance. Under this motivation, this paper proposes an improved covariant local feature detector by fusing feature information obtained from another detector, which is served as a confidence to guide the voting procedure when converting the predicted transformations into a meaningful score map for keypoint detection. In this way, the fused information can enhance the features that are considered to be good and weaken those unstable features. The proposed method is evaluated on four widely used benchmarks and consistent performance improvement over previous works is observed.

Introduction

Local feature detection plays a vital role in many computer vision applications, such as object recognition [1], 3D reconstruction [2], [3], image retrieval [4], face recognition [5], [6] and so on. It has been an active research topic in the past decades, and many excellent local feature detectors have been proposed. In the early days, feature detection is primarily relied on handcrafting, and various local feature detectors have been developed based on different visual structures, such as corners and blobs. To achieve scale invariance, most of the handcrafted features resort to build an image scale space by progressively applying Gaussian smoothing and localize keypoints with maxima responses in the scale space as the detected features along with their scales. For corner detectors, the most famous feature detectors are Harris [7], FAST [8], and their variants (ORB [9], BRISK [10], etc.). While for the blob structures in image, determinant of hessian matrix [11] and response of LoG (Laplacian of Gaussian) filters [1], [12] have been proposed to localize blob-like features. Although these methods have explicit physical meanings, due to the inaccurate of artificially defined response functions and the complexity of the images, many undesired feature points are often detected, which in turn affects the performance of subsequent feature matching.

With the development of deep learning, recent progress has also been made in learning detectors. Lenc and Vedaldi [13] proposed CovDet to cast feature detection as a regression problem, which trains a local transformation predictor instead of learning a score map indicating the probability of local features. Zhang et al. [14] extended CovDet by introducing the concepts of ǣstandard patchǥ and ǣcanonical featureǥ and proposed DT-CovDet, towards obtaining more robust local features. Although these deep learning based methods outperform traditional handcrafted ones, they are still based on one specific metric (here is the prediction from nearby pixels) to determine whether a point is a keypoint or not. Due to the complexity of natural images, single information about local image structures is usually not enough for detecting reliable keypoints across different images with high repeatability.

To alleviate this problem, we propose a method by fusing DT-CovDet that detects keypoints by transforming nearby pixels, with other keypoint detectors that detect keypoints by measuring the likelihood of pixels. On one hand, many existing detectors are built up on well established physical properties of keypoint, which is able to provide a coarse score about the likelihood of a pixel being a keypoint. On the other hand, the DT-CovDet is learned from data by imposing nearby pixels moving towards the underlying keypoints. These kinds of information are complementary and could be combined to improve the keypoint detection performance. In other words, considering both kinds of feature information at the same time will make the obtained local features more robust, i.e., enhancing the detection of consistent features while suppressing those with inconsistent context. The proposed framework is illustrated in Fig. 1. We have evaluated the proposed method on three traditional datasets (EF [15], VGGe [16], Webcam [17]) and the newly proposed large-scale benchmark, HPatches [18]. Consistent performance improvement on all these tested datasets is observed.

The following of this paper is organized as: Section 2 describes the related work. Section 3 elaborates the proposed method. Experiments are demonstrated in Section 4, while Section 5 concludes the paper.