Method and system for image registrations

• US 10,013,765 B2
• Filed: 08/19/2016
• Issued: 07/03/2018
• Est. Priority Date: 08/19/2016
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Ramalingam, Srikumar, Taguchi, Yuichi, Sankaran, Bharath
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US15/241,118
• Publication Number:
  US 20180053293A1
• Time in Patent Office: 683 Days
• Field of Search: 382100, 382104, 382106, 382159, 382192, 382173, 382180, 382103, 382181, 382224, 382128, 382195, 382201, 382199, 382168, 382170, 382197, 382203, 382255
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching

Abstract

An image registrations includes determining a first binary descriptor of a first key point in a first image, determining a second binary descriptor of a second key point in a second image, determining a weighted Hamming distance between the first binary descriptor and the second binary descriptor, and registering the first key point with the second key point when the weighted Hamming distance is below a noise threshold. At least one element in the first or the second binary descriptor is a result of a comparison of a difference between intensities of at least two pixels of the first or the second image with a threshold. At least two weights of the weighted Hamming distance for comparing at least two elements of the first or the second binary descriptors are different.

Claims

1. A method for image registrations, comprising:

determining a first binary descriptor of a first key point in a first image, wherein at least one element in the first binary descriptor is a result of a comparison of a difference between intensities of at least two pixels of the first image with a threshold;

determining a second binary descriptor of a second key point in a second image, wherein at least one element in the second binary descriptor is a result of a comparison of a difference between intensities of at least two pixels of the second image with the threshold;

determining a weighted distance between the first binary descriptor and the second binary descriptor, wherein at least two weights of the weighted distance for comparing at least two elements of the first and the second binary descriptors are different;

registering the first key point with the second key point when the weighted distance is below a noise threshold, wherein at least some steps of the method are performed using a processor; and

executing a computer vision action based on the image registrations, wherein the computer vision action includes one or combination of visual odometry, 3D reconstruction, object detection, object recognition, sensor fusion, and image-based localization.

2. The method of claim 1, further comprising:

selecting a first set of key points in the first image and a second set of key points in the second image using at least one or combination of HARRIS, FAST, or SURF keypoint detectors; and

repeating the determining steps for each combination of pairs of key points from the first and the second set to register the pairs of key points with the weighted distance below the noise threshold.

3. The method of claim 2, further comprising:

selecting a first patch of the first image that includes the first key point;

selecting a second patch of the second image that includes the second key point;

determining the first binary descriptors by comparing intensities of at least some pixels from the first patch; and

determining the second binary descriptors by comparing intensities of at least some pixels from the second patch.

4. The method of claim 1, wherein the weighted distance is a weighted Hamming distance H_w determined according to

5. The method of claim 4, wherein N equals 256, and wherein the threshold is greater than zero.

6. The method of claim 1, wherein an i^th element b_i, of the first or the second binary descriptors is determined according to

7. The method of claim 1, further comprising:

acquiring the first and the second images captured with a camera at different points of time; and

selecting, from a memory operatively connected with the processor, at least one of the threshold, the noise threshold, and the weights of the weighted Hamming distance based on a function the points of time.

8. The method of claim 7, wherein the memory stores different values of at least one of the threshold, the noise threshold, and the weights of the weighted Hamming distance for images captured at different times of a date, at different seasons of the year, at different lighting conditions, and for different types of scenes.

9. The method of claim 1, wherein the weights of the weighted Hamming distance are learned from training data using a max-margin network learning.

10. The method of claim 9, wherein the threshold is learned by enumerating over different integer values during the max-margin network learning.

11. A system for image registrations, comprising:

an input interface for acquiring a first image and a second image; and

a process for determining a first binary descriptor of a first key point in a first image, determining a second binary descriptor of a second key point in a second image, determining a weighted Hamming distance between the first binary descriptor and the second binary descriptor, and registering the first key point with the second key point when the weight Hamming distance is below a noise threshold, wherein at least one element in the first or the second binary descriptor is a result of a comparison of a difference between intensities of at least two pixels of the first or the second image with a non-zero threshold, and wherein at least two weights of the weighted Hamming distance for comparison at least two elements of the first or the second binary descriptors are different; and

an output interface for executing a computer vision action based on the image registrations, wherein the computer vision action includes one or combination of visual odometry, 3D reconstruction, object detection, object recognition, sensor fusion, and image-based localization.

12. A camera including the system of claim 11.

13. The system of claim 11, further comprising:

a memory to store different values of at least one of the threshold, the noise threshold, and the weights of the weighted Hamming distance for images captured at different times of a date, at different seasons of the year, at different lighting conditions, and for different types of scenes.

14. The system of claim 13, wherein the processor is configured to

determine points of time of capturing the first and the second images;

and

select, from the memory, at least one of the threshold, the noise threshold, and the weights of the weighted Hamming distance based on a function the points of time.

15. The system of claim 11, wherein the weighted Hamming distance H_w is determined according to

16. The system of claim 15, wherein N equals 256.

17. The system of claim 11, wherein an i^th element b_i, of the first or the second binary descriptors is determined according to

18. A non-transitory computer readable storage medium embodied thereon a program executable by a processor for performing a method, the method comprising:

determining a first binary descriptor of a first key point in a first image, wherein at least one element in the first binary descriptor is a result of a comparison of a difference between intensities of at least two pixels of the first image with a threshold;

determining a second binary descriptor of a second key point in a second image, wherein at least one element in the second binary descriptor is a result of a comparison of a difference between intensities of at least two pixels of the second image with the threshold;

determining a weighted Hamming distance between the first binary descriptor and the second binary descriptor, wherein at least two weights of the weighted Hamming distance for comparing at least two elements of the first and the second binary descriptors are different;

registering the first key point with the second key point when the weighted Hamming distance is below a noise threshold; and

executing a computer vision action based on the image registrations, wherein the computer vision action includes one or combination of visual odometry, 3D reconstruction, object detection, object recognition, sensor fusion, and image-based localization.