UNIFIED FRAMEWORK FOR PRECISE VISION-AIDED NAVIGATION

• US 20120206596A1
• Filed: 04/19/2012
• Published: 08/16/2012
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: SAMARASEKERA, SUPUN, KUMAR, RAKESH, OSKIPER, TARAGAY, ZHU, ZHIWEI, NARODITSKY, OLEG, SAWHNEY, HARPREET
• Granted Patent: US 9,121,713 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A system and method for efficiently locating in 3D an object of interest in a target scene using video information captured by a plurality of cameras. The system and method provide for multi-camera visual odometry wherein pose estimates are generated for each camera by all of the cameras in the multi-camera configuration. Furthermore, the system and method can locate and identify salient landmarks in the target scene using any of the cameras in the multi-camera configuration and compare the identified landmark against a database of previously identified landmarks. In addition, the system and method provide for the integration of video-based pose estimations with position measurement data captured by one or more secondary measurement sensors, such as, for example, Inertial Measurement Units (IMUs) and Global Positioning System (GPS) units.

Claims

1. A computer implemented landmark recognition method comprising:

capturing a plurality of images where the plurality of images comprises a view of one or more landmarks obtained by each of a plurality of cameras, where each camera in the plurality of cameras has different position measurement data comprising location, depth and pose as compared to position measurement data of each other camera;

computing a landmark snapshot for each landmark of the one or more landmarks in the plurality of images, the landmark snapshot comprising a scale identifier for the landmark combined with a spatial configuration for the landmark; and

comparing between the landmark snapshot for each of the one or more landmarks in the plurality of images to identify corresponding landmarks in the plurality of images.

2. The method of claim 1, wherein a set of pose hypotheses for each camera in the plurality of cameras are generated based on randomly selected 3 point correspondences over two views of a corresponding camera in the plurality of cameras.

3. The method of claim 1 wherein the scale identifier is a histogram of oriented gradients (HOG) descriptor.

4. The method of claim 1 wherein the scale identifier is a fixed scale identifier.

5. The method of claim 1 wherein the scale identifier is a depth-dependent scale identifier.

6. The method of claim 1 further comprising:

scoring a set of pose hypotheses generated for each camera based on a comparison within a local dataset of point correspondences for each camera within the plurality of cameras to produce a set of intra-camera scores; and

identifying an optimized video-based pose hypothesis for each camera based on an evaluation of a set of global scores generated by transferring the set of pose hypotheses generated for each camera to other cameras in the system and producing a set of inter-camera scores.

7. The method of claim 6, further comprising integrating the pose hypotheses with the position measurement data.

8. The method of claim 7 wherein the integrating comprising performing a Kalman filtering operation on the position measurement data and the pose hypotheses.

9. The method of claim 6, further comprising determining positional measurement data using an inertial measurement unit (IMU).

10. The method of claim 6, further comprising determining positional measurement data using a global positioning system (GPS).

11. The method of claim 6, further comprising refining the identified landmark matches by estimating a fundamental matrix using a RANSAC method, and based on the estimated fundamental matrix discarding one or more matches that produce a residual that is higher than a predefined threshold value.

12. The method of claim 11, wherein the set of pose hypotheses are generated based on randomly selected 5 point correspondences over three views of each camera within the plurality of cameras using local camera data corresponding to each camera.

13. The method of claim 12, wherein the set of pose hypotheses are generated based on randomly selected 5 point correspondences over three views using data from the plurality of cameras.

14. The method of claim 1 wherein the comparing between the landmark snapshot further comprises:

determining correspondence of two-dimensional features in the landmark snapshot for each of the one or more landmarks in the plurality of images for reconstructing three-dimensional coordinates for the two-dimensional features;

determining scale for each scale identifier that is proportional to the distance between each camera within the plurality of cameras and the landmark snapshot associated with each camera;

comparing a cosine similarity score of the scale identifier to determine if one or more landmarks correspond.

15. The method of claim 1 further comprising:

pruning the landmark snapshot for each of the one or more landmarks to identify a refined set of candidate landmark snapshots; and

computing a histogram vector corresponding to the spatial distribution of the landmarks in each landmark snapshot.

16. The method of claim 15 further comprising ranking the candidate landmark according to time of capture.