Unified framework for precise vision-aided navigation

• US 9,121,713 B2
• Filed: 04/19/2012
• Issued: 09/01/2015
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Kumar, Rakesh, Oskiper, Taragay, Zhu, Zhiwei, Naroditsky, Oleg, Sawhney, Harpreet
• Primary Examiner(s): Walsh, John B
• Application Number:
  US13/451,037
• Publication Number:
  US 20120206596A1
• Time in Patent Office: 1,230 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A system and method for efficiently locating in 3D an object of interest in a target scene using video information captured by a plurality of cameras. The system and method provide for multi-camera visual odometry wherein pose estimates are generated for each camera by all of the cameras in the multi-camera configuration. Furthermore, the system and method can locate and identify salient landmarks in the target scene using any of the cameras in the multi-camera configuration and compare the identified landmark against a database of previously identified landmarks. In addition, the system and method provide for the integration of video-based pose estimations with position measurement data captured by one or more secondary measurement sensors, such as, for example, Inertial Measurement Units (IMUs) and Global Positioning System (GPS) units.

Claims

1. A computer implemented landmark recognition method comprising:

capturing a plurality of images where the plurality of images comprises a view of one or more landmarks obtained by each of a plurality of cameras, where each camera in the plurality of cameras has different position measurement data comprising location, depth and pose as compared to position measurement data of each other camera;

computing a landmark snapshot for each landmark of the one or more landmarks in the plurality of images, the landmark snapshot comprising a scale identifier for the landmark combined with a spatial configuration for the landmark, wherein the scale identifier comprises a histogram of oriented gradients (HOG) descriptor; and

comparing between the landmark snapshot for each of the one or more landmarks in the plurality of images to identify corresponding landmarks in the plurality of images.

2. The method of claim 1, wherein a set of pose hypotheses for each camera in the plurality of cameras are generated based on randomly selected three point correspondences over two views of a corresponding camera in the plurality of cameras.

3. The method of claim 1 wherein the scale identifier is a fixed scale identifier.

4. The method of claim 1 wherein the scale identifier is a depth-dependent scale identifier.

5. The method of claim 1 further comprising:

scoring a set of pose hypotheses generated for each camera based on a comparison within a local dataset of point correspondences for each camera within the plurality of cameras to produce a set of intra-camera scores; and

identifying an optimized video-based pose hypothesis for each camera based on an evaluation of a set of global scores generated by transferring the set of pose hypotheses generated for each camera to other cameras and producing a set of inter-camera scores.

6. The method of claim 5, further comprising integrating the set of pose hypotheses with the position measurement data.

7. The method of claim 6 wherein the integrating comprises performing a Kalman filtering operation on the position measurement data and the set of pose hypotheses.

8. The method of claim 5, wherein the set of pose hypotheses are generated using local camera data corresponding to each camera.

9. The method of claim 8, wherein the set of pose hypotheses are generated based on randomly selected five point correspondences over views of each camera within the plurality of cameras.

10. The method of claim 1, further comprising determining positional measurement data using a global positioning system (GPS).

11. The method of claim 1, further comprising refining the corresponding landmarks identified in the plurality of images by estimating a fundamental matrix using a RANSAC method, and based on the fundamental matrix discarding one or more of the corresponding landmarks that produce a residual that is higher than a predefined threshold value.

12. The method of claim 1, further comprising determining positional measurement data using an inertial measurement unit (IMU).

13. The method of claim 1 further comprising:

pruning the landmark snapshot for each of the one or more landmarks to identify a refined set of candidate landmark snapshots.

14. The method of claim 13 further comprising ranking each of the refined set of candidate landmark snapshots according to time of capture.

15. The method of claim 1 wherein the comparing between the landmark snapshot further comprises:

determining a correspondence of two-dimensional features in the landmark snapshot for each of the one or more landmarks in the plurality of images;

determining a scale for each scale identifier that is proportional to a distance between each camera within the plurality of cameras and the landmark snapshot associated with each camera; and

comparing a similarity score of the scale identifier to determine if one or more landmarks correspond.

16. An apparatus for landmark recognition comprising:

a camera cluster that captures a plurality of images where the plurality of images comprises a view of one or more landmarks, wherein each camera in the camera cluster has different position measurement data comprising location, depth and pose as compared to position measurement data of each other camera;

a landmark database that stores a landmark snapshot for each landmark of the one or more landmarks in the plurality of images, the landmark snapshot comprising a scale identifier for the landmark associated with a spatial configuration for the landmark, wherein the scale identifier comprises a histogram of oriented gradients (HOG) descriptor; and

a landmark matcher that compares between the landmark snapshot for each of the one or more landmarks in the plurality of images to identify corresponding landmarks in the plurality of images.

17. The apparatus of claim 16 wherein the landmark matcher further:

scores a set of pose hypotheses generated for each camera based on a comparison within a local dataset of point correspondences for each camera within the camera cluster to produce a set of intra-camera scores; and

identifies an optimized video-based pose hypothesis for each camera based on an evaluation of a set of global scores generated by transferring the set of pose hypotheses generated for each camera to other cameras and producing a set of inter-camera scores.

18. The apparatus of claim 16, wherein the landmark matcher further:

determines a correspondence of two-dimensional features in the landmark snapshot for each of the one or more landmarks in the plurality of images;

determines a scale for each scale identifier that is proportional to a distance between each camera within the camera cluster and the landmark snapshot associated with each camera; and

compares a similarity score of the scale identifier to determine if one or more landmarks correspond.

19. The apparatus of claim 16, further comprising a landmark pruner that prunes the landmark snapshot for each of the one or more landmarks to identify a refined set of candidate landmark snapshots.

20. The apparatus of claim 19, wherein the landmark pruner is operative to perform at least one of:

computing a histogram vector corresponding to a spatial distribution of the landmarks in each landmark snapshot, and ranking the refined set of candidate landmark snapshots according to time of capture.