Method and system for real-time visual odometry

• US 8,213,706 B2
• Filed: 04/22/2008
• Issued: 07/03/2012
• Est. Priority Date: 04/22/2008
• Status: Expired due to Fees

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Krishnaswamy, Kailash, Susca, Sara, McCroskey, Robert C.
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): WOLDEMARIAM, AKLILU K
• Application Number:
  US12/107,722
• Publication Number:
  US 20090263009A1
• Time in Patent Office: 1,533 Days
• Field of Search: 382/154, 700/250, 700/245, 700/258, 700/263
• US Class Current: 382/154
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/35 using statistical methods

Abstract

A method for real-time visual odometry comprises capturing a first three-dimensional image of a location at a first time, capturing a second three-dimensional image of the location at a second time that is later than the first time, and extracting one or more features and their descriptors from each of the first and second three-dimensional images. One or more features from the first three-dimensional image are then matched with one or more features from the second three-dimensional image. The method further comprises determining changes in rotation and translation between the first and second three-dimensional images from the first time to the second time using a random sample consensus (RANSAC) process and a unique iterative refinement technique.

Claims

1. A method for real-time visual odometry, the method comprising:

capturing a first three-dimensional image of a location at a first time;

capturing a second three-dimensional image of the location at a second time that is later than the first time;

extracting one or more features and their descriptors from each of the first and second three-dimensional images;

matching one or more features from the first three-dimensional image with one or more features from the second three-dimensional image; and

determining a change in rotation and a change in translation between the first and second three-dimensional images from the first time to the second time using a random sample consensus (RANSAC) process and an iterative refinement technique, wherein the iterative refinement technique comprises;

receiving a first set of inliers and changes in rotation and translation for the first set of inliers outputted from the RANSAC process;

scoring the changes in rotation and translation for the first set of inliers;

operating on the first set of inliers using a least squares method in a first joint trial to produce a second set of inliers during a first iteration;

computing changes in rotation and translation for the second set of inliers from the first joint trial;

scoring the changes in rotation and translation for the second set of inliers from the first joint trial; and

comparing the score of the changes in rotation and translation for the first set of inliers with the score of the changes in rotation and translation for the second set of inliers to determine appropriate outputs during the first iteration.

2. The method of claim 1, wherein comparing the score of the changes in rotation and translation for the first set of inliers with the score of the changes in rotation and translation for the second set of inliers further comprises:

outputting the changes in rotation and translation for the first set of inliers when the score of the changes in rotation and translation for the first set of inliers is greater than the score of the changes in rotation and translation for the second set of inliers; and

outputting the changes in rotation and translation for the second set of inliers when the score of the changes in rotation and translation for the first set of inliers is equal to the score of the changes in rotation and translation for the second set of inliers.

3. The method of claim 2, wherein if the score of the changes in rotation and translation for the first set of inliers is not greater than or equal to the score of the changes in rotation and translation for the second set of inliers, the method further comprises:

operating on the second set of inliers using a least squares method in a second joint trial to produce a third set of inliers during a second iteration;

computing changes in rotation and translation for the third set of inliers from the second joint trial;

scoring the changes in rotation and translation for the third set of inliers from the second joint trial; and

comparing the score of the changes in rotation and translation for the second set of inliers with the score of the changes in rotation and translation for the third set of inliers to determine appropriate outputs during the second iteration.

4. The method of claim 3, wherein comparing the score of the changes in rotation and translation for the second set of inliers with the score of the changes in rotation and translation for the third set of inliers further comprises:

outputting the changes in rotation and translation for the second set of inliers when the score of the changes in rotation and translation for the second set of inliers is greater than the score of the changes in rotation and translation for the third set of inliers; and

outputting the changes in rotation and translation for the third set of inliers when the score of the changes in rotation and translation for the second set of inliers is equal to the score of the changes in rotation and translation for the third set of inliers.

5. The method of claim 4, wherein if the score of the changes in rotation and translation for the second set of inliers is not greater than or equal to the score of the changes in rotation and translation for the third set of inliers, the method further comprises:

operating on the third set of inliers using a least squares method in a third joint trial to produce a fourth set of inliers during a third iteration;

computing changes in rotation and translation for the fourth set of inliers from the third joint trial; and

outputting the changes in rotation and translation for the fourth set of inliers.

6. The method of claim 1, wherein the first and second three-dimensional images are captured with a stereo camera.

7. The method of claim 1, wherein the first and second three-dimensional images each comprise a composite of a left image frame and a right image frame.

8. The method of claim 7, wherein the left image frame and the right image frame are projected onto a common image surface using an image rectifier.

9. The method of claim 7, further comprising extracting one or more features and their descriptors from the left image frame using a first processor.

10. The method of claim 9, further comprising extracting one or more features and their descriptors from the right image frame using a second processor in parallel with the first processor.

11. The method of claim 10, further comprising matching one or more features from the left image frame with one or more features from the right image frame.

12. The method of claim 11, further comprising matching one or more features from the left image frame and the right image frame, respectively, with one or more features from a subsequent left image frame and right image frame.

13. The method of claim 12, further comprising constructing a three-dimensional data point cloud.

14. The method of claim 1, wherein the first and second three-dimensional images are captured with an image sensor device comprising a light detection and ranging device, a radar device, or a sonar device.

15. The method of claim 1, wherein the features are matched by employing an epipolar constraint.

16. A system for real-time visual odometry, comprising:

an image sensor device configured to capture consecutive three-dimensional images each comprising a left image frame and a right image frame;

an image rectifier configured to project the left image frame and the right image frame onto a common image surface;

a first processor configured to extract one or more features and their descriptors from the left image frame;

a second processor in parallel with the first processor and configured to extract one or more features and their descriptors from the right image frame;

a first feature matching unit configured to match one or more features from the left image frame with one or more features from the right image frame;

a second feature matching unit configured to match one or more features from the left image frame and the right image frame, respectively, with one or more features from a subsequent left image frame and right image frame; and

a robust pose estimator configured to determine a change in rotation and a change in translation between the consecutive three-dimensional images, wherein the robust pose estimator comprises program code for a random sample consensus (RANSAC) process and a refinement technique, wherein the refinement technique comprises;

program code to receive a first set of inliers and changes in rotation and translation for the first set of inliers outputted from the RANSAC process;

program code to score the changes in rotation and translation for the first set of inliers;

program code to operate on the first set of inliers using a least squares method in a first joint trial to produce a second set of inliers;

program code to compute changes in rotation and translation for the second set of inliers from the first joint trial; and

program code to score the changes in rotation and translation for the second set of inliers from the first joint trial.

17. The system of claim 16, wherein the image sensor device comprises a stereo camera.

18. The system of claim 16, wherein the refinement technique further comprises:

program code to compare the score of the changes in rotation and translation for the first set of inliers with the score of the changes in rotation and translation for the second set of inliers to determine appropriate outputs during one or more iterations.

19. A computer program product, comprising:

a non-transitory computer readable medium having instructions stored thereon for implementing a method for real-time visual odometry using a random sample consensus (RANSAC) process and an iterative refinement technique, wherein the iterative refinement technique comprises;

receiving a first set of inliers and changes in rotation and translation for the first set of inliers outputted from the RANSAC process;

scoring the changes in rotation and translation for the first set of inliers;

operating on the first set of inliers using a least squares method in a first joint trial to produce a second set of inliers;

computing changes in rotation and translation for the second set of inliers from the first joint trial;

scoring the changes in rotation and translation for the second set of inliers from the first joint trial; and

comparing the score of the changes in rotation and translation for the first set of inliers with the score of the changes in rotation and translation for the second set of inliers to determine appropriate outputs during one or more iterations.

20. The computer program product of claim 19, wherein comparing the score of the changes in rotation and translation for the first set of inliers with the score of the changes in rotation and translation for the second set of inliers further comprises:

outputting the changes in rotation and translation for the first set of inliers when the score of the changes in rotation and translation for the first set of inliers is greater than the score of the changes in rotation and translation for the second set of inliers; and

outputting the changes in rotation and translation for the second set of inliers when the score of the changes in rotation and translation for the first set of inliers is equal to the score of the changes in rotation and translation for the second set of inliers.