FEATURE DESCRIPTOR MATCHING

• US 20190318502A1
• Filed: 04/11/2019
• Published: 10/17/2019
• Est. Priority Date: 04/12/2018
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: He, Kun, Lu, Yan
• Granted Patent: US 10,997,746 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 3/10 Selection of transformation methods according to the characteristics of the input images
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Feature descriptor matching described herein may include receiving a first input image and a second input image. A feature detector may detect features from the first and second input images. A descriptor extractor may learn local feature descriptors from the features of the first and second input images based on a feature descriptor matching model trained using a ground truth data set. The descriptor extractor may determine a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors. The descriptor matcher may generate a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network.

Claims

1. A system for feature descriptor matching, comprising:

a memory receiving a first input image and a second input image;

a feature detector detecting a first set of features from the first input image and a second set of features from the second input image;

a descriptor extractor learning a first set of local feature descriptors from the first set of features of the first input image and a second set of local feature descriptors from the second set of features of the second input image based on a feature descriptor matching model trained using a ground truth data set including a first ground truth image and a second ground truth image;

the descriptor extractor determining a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors; and

a descriptor matcher generating a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network (CNN).

2. The system for feature descriptor matching of claim 1, wherein the first set of local feature descriptors or the second set of local feature descriptors includes a vector representation of a corresponding image patch.

3. The system for feature descriptor matching of claim 1, wherein the first set of local feature descriptors or the second set of local feature descriptors includes a binary descriptor or a real-valued descriptor.

4. The system for feature descriptor matching of claim 3, wherein the descriptor matcher generates the geometric transformation further based on an amount of computing resources available within the system for feature descriptor matching and one of:

binary descriptors of the first set of local feature descriptors and binary descriptors of the second set of local feature descriptors;

or

real-valued descriptors of the first set of local feature descriptors and real-valued descriptors of the second set of local feature descriptors.

5. The system for feature descriptor matching of claim 1, wherein the first ground truth image and the second ground truth image of the ground truth data set are hard negatives with no matching features.

6. The system for feature descriptor matching of claim 1, wherein a geometric alignment is performed on the first ground truth image and the second ground truth image prior to training the feature descriptor matching model using the first ground truth image and the second ground truth image.

7. The system for feature descriptor matching of claim 6, wherein the geometric alignment is performed based on a spatial transformer network.

8. The system for feature descriptor matching of claim 1, wherein the descriptor extractor performs label mining based on clustering while learning the first set of local feature descriptors or the second set of local feature descriptors.

9. The system for feature descriptor matching of claim 8, wherein one or more image patches are clustered based on an inter-cluster distance from other image patches.

10. The system for feature descriptor matching of claim 1, wherein the feature descriptor matching model or the CNN is trained based on stochastic gradient descent (SGD).

11. A method for feature descriptor matching, comprising:

receiving a first input image and a second input image;

detecting a first set of features from the first input image and a second set of features from the second input image;

learning a first set of local feature descriptors from the first set of features of the first input image and a second set of local feature descriptors from the second set of features of the second input image based on a feature descriptor matching model trained using a ground truth data set including a first ground truth image and a second ground truth image;

determining a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors; and

generating a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network (CNN).

12. The method for feature descriptor matching of claim 11, wherein the first set of local feature descriptors or the second set of local feature descriptors includes a binary descriptor or a real-valued descriptor.

13. The method for feature descriptor matching of claim 12, further comprising generating the geometric transformation further based on an amount of computing resources available within a system for feature descriptor matching and one of:

binary descriptors of the first set of local feature descriptors and binary descriptors of the second set of local feature descriptors;

or

real-valued descriptors of the first set of local feature descriptors and real-valued descriptors of the second set of local feature descriptors.

14. The method for feature descriptor matching of claim 11, wherein the first ground truth image and the second ground truth image of the ground truth data set are hard negatives with no matching features.

15. The method for feature descriptor matching of claim 11, further comprising performing a geometric alignment on the first ground truth image and the second ground truth image prior to training the feature descriptor matching model using the first ground truth image and the second ground truth image.

16. The method for feature descriptor matching of claim 15, wherein the geometric alignment is performed based on a spatial transformer network.

17. A system for feature descriptor matching, comprising:

a memory receiving a first input image and a second input image;

a feature detector detecting a first set of features from the first input image and a second set of features from the second input image;

a descriptor extractor learning a first set of local feature descriptors from the first set of features of the first input image and a second set of local feature descriptors from the second set of features of the second input image based on a feature descriptor matching model trained using a ground truth data set including a first ground truth image and a second ground truth image, wherein the first set of local feature descriptors and the second set of local feature descriptors include a binary descriptor or a real-valued descriptor;

the descriptor extractor determining a listwise mean average precision (mAP) rank of a pool of candidate image patches from the second input image with respect to a queried image patch from the first input image based on the feature descriptor matching model, the first set of local feature descriptors, and the second set of local feature descriptors; and

a descriptor matcher generating a geometric transformation between the first input image and the second input image based on the listwise mAP and a convolutional neural network (CNN).

18. The system for feature descriptor matching of claim 17, wherein the descriptor matcher generates the geometric transformation further based on an amount of computing resources available within the system for feature descriptor matching and one of:

binary descriptors of the first set of local feature descriptors and binary descriptors of the second set of local feature descriptors;

or

real-valued descriptors of the first set of local feature descriptors and real-valued descriptors of the second set of local feature descriptors.

19. The system for feature descriptor matching of claim 17, wherein the first ground truth image and the second ground truth image of the ground truth data set are hard negatives with no matching features.

20. The system for feature descriptor matching of claim 17, wherein geometric alignment is performed on the first ground truth image and the second ground truth image prior to training the feature descriptor matching model using the first ground truth image and the second ground truth image.