UNIFIED FRAMEWORK FOR PRECISE VISION-AIDED NAVIGATION

• US 20080167814A1
• Filed: 12/03/2007
• Published: 07/10/2008
• Est. Priority Date: 12/01/2006
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Sawhney, Harpreet, Samarasekera, Supun, Kumar, Rakesh, Zhu, Zhiwei, Naroditsky, Oleg, Oskiper, Taragay
• Granted Patent: US 8,174,568 B2
• Time in Patent Office: Days
• US Class Current: 701/213
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A system and method for efficiently locating in 3D an object of interest in a target scene using video information captured by a plurality of cameras. The system and method provide for multi-camera visual odometry wherein pose estimates are generated for each camera by all of the cameras in the multi-camera configuration. Furthermore, the system and method can locate and identify salient landmarks in the target scene using any of the cameras in the multi-camera configuration and compare the identified landmark against a database of previously identified landmarks. In addition, the system and method provide for the integration of video-based pose estimations with position measurement data captured by one or more secondary measurement sensors, such as, for example, Inertial Measurement Units (IMUs) and Global Positioning System (GPS) units.

Claims

1. A method for performing multi-camera visual odometry across a system including a plurality of cameras, wherein each camera comprises a local dataset, comprising the steps of:

for each camera, generating a set of pose hypotheses;

for each camera, scoring the set of pose hypotheses based on a comparison with the local dataset to produce a set of intra-camera scores;

transferring the set of pose hypotheses to the other cameras in the system;

for each camera, scoring the set of transferred pose hypotheses based on the local dataset to produce a set of inter-camera scores;

combining the set of intra-camera scores and the one or more sets of inter-camera scores to produce a set of global scores; and

identifying an optimized video-based pose hypothesis based on an evaluation of the set of global scores.

2. The method of claim 1 further comprising the step of refining the video-based pose hypothesis across all of the cameras in the system to produce a refined pose estimation for each camera.

3. The method of claim 1, wherein the set of pose hypotheses are generated based on randomly selected 5 point correspondences over three views using local camera data.

4. The method of claim 3, wherein the set of pose hypotheses are generated based on randomly selected 5 point correspondences over three views using data from the plurality of cameras.

5. The method of claim 1, wherein the set of pose hypotheses are generated based on randomly selected 3 point correspondences over two views.

6. The method of claim 5, wherein the set of pose hypotheses are generated based on randomly selected 3 point correspondences over two views using data from the plurality of cameras.

7. The method of claim 1, wherein the step of transferring the set of pose hypotheses to the other cameras in the system further comprises transforming each camera'"'"'s pose relative to the other cameras in the system.

8. A navigation system comprising:

a plurality of cameras configured to capture video related to a target scene;

a visual odometry module configured to;

receive the video from each of the plurality of cameras,

detect one or more features of the target scene,

track the one or more features to produce one or more feature tracks, and

generate an optimized video-based pose estimation based on the one or more feature tracks,

an integration module communicatively connected to the visual odometry module and one or more secondary measurement sensors configured to capture position measurement data related to the target scene, the integration module configured to;

integrate the video-based pose estimation and the position measurement data, and

generate a filtered pose estimation; and

a global landmark recognition module configured to;

receive a video snapshot related to the target scene from the plurality of cameras,

extract one or more landmarks from the video snapshot;

match the one or more extracted landmarks with historical landmark information stored in a landmark database to produce landmark matching information,

compute a landmark-based pose estimation based at least in part on the landmark matching information, and

compute a final pose estimation based at least on a comparison of the video-based pose estimation and the landmark-based pose estimation.

9. The system of claim 8, wherein the integration module comprises a Kalman filter.

10. The system of claim 8, wherein the one or more measurement sensors comprise an inertial measurement unit (IMU).

11. The system of claim 8, wherein the one or more measurement sensors comprise a global positioning system (GPS).

12. The system of claim 8, further comprising the steps of:

associating the extracted landmark with a HOG descriptor to produce a landmark snapshot; and

matching the extracted landmark snapshot with historical landmark snapshots stored in the landmark database.

13. A landmark recognition method comprising the steps of:

receiving a video snapshot comprising a pair of stereo images related to a target scene,

extracting a set of feature points from each stereo image,

identifying correspondences among the extracted set of feature points between the pair of stereo images,

generating 3D coordinates for each feature point, wherein the 3D coordinates of the set of feature points comprises a landmark,

computing a histogram of oriented gradients (HOG) descriptor based on at least a portion of the stereo image pair,

associating the landmark with the HOG descriptor to produce a landmark snapshot, and

matching the landmark snapshot with one or more historical landmarks stored in a database.

14. The method of claim 13, wherein the landmark snapshot further comprises spatial configuration data.

15. The method of claim 13, wherein the spatial configuration data comprises 2D coordinate and 3D coordinate data related to the feature points.

16. The method of claim 13, wherein the HOG descriptor comprises a depth-dependent scale.

17. A landmark recognition method comprising the steps of:

capturing a first image comprising a first set of landmarks, wherein each landmark is associated with a unique HOG descriptor;

capturing a second image comprising a second set of landmarks wherein each landmark is associated with a unique HOG descriptor; and

searching the second set of landmarks in the second image to identify corresponding landmark matches in the first set of landmarks, based on a comparison of the HOG descriptors.

18. The method of claim 17, further comprising the step of refining the identified landmark matches by estimating a fundamental matrix using a RANSAC method, and discarding one or more matches that produce a residual based on the estimated fundamental matrix that is higher than a predefined threshold value.

19. A method for determining the location of an object:

receiving new location information relating to an object;

extracting a landmark snapshot from the new location information;

associating the landmark snapshot with a HOG descriptor;

generating a geo-spatial search region based at least on the new location information;

selecting an initial set of candidate landmark snapshots based on the geo-spatial search region;

pruning the initial set of candidate landmark snapshots to identify a refined set of candidate landmark snapshots;

identifying an optimized candidate landmark snapshot based on a comparison of the HOG descriptor of the extracted landmark snapshot and the HOG descriptors of the refined set of candidate landmark snapshots; and

determining the location of the object based at least in part on the optimized candidate landmark snapshot.

20. The method of claim 19, wherein the new location information comprises estimated 3D coordinates.