Mapping and tracking system with features in three-dimensional space

• US 10,027,952 B2
• Filed: 08/03/2012
• Issued: 07/17/2018
• Est. Priority Date: 08/04/2011
• Status: Active Grant

• Current Assignee: TRX Systems, Inc.
• Original Assignee: TRX Systems, Inc.
• Inventors: Karvounis, John George
• Primary Examiner(s): TORRENTE, RICHARD T
• Application Number:
  US13/566,956
• Publication Number:
  US 20150304634A1
• Time in Patent Office: 2,174 Days
• Field of Search: 348 46
• CPC Class Codes: G06F 18/21355 nonlinear criteria, e.g. embedding a manifold in a Euclidean space
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20088 Trinocular vision calculations; trifocal tensor
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 2201/11 Technique with transformation invariance effect
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

LK-SURF, Robust Kalman Filter, HAR-SLAM, and Landmark Promotion SLAM methods are disclosed. LK-SURF is an image processing technique that combines Lucas-Kanade feature tracking with Speeded-Up Robust Features to perform spatial and temporal tracking using stereo images to produce 3D features can be tracked and identified. The Robust Kalman Filter is an extension of the Kalman Filter algorithm that improves the ability to remove erroneous observations using Principal Component Analysis and the X84 outlier rejection rule. Hierarchical Active Ripple SLAM is a new SLAM architecture that breaks the traditional state space of SLAM into a chain of smaller state spaces, allowing multiple tracked objects, multiple sensors, and multiple updates to occur in linear time with linear storage with respect to the number of tracked objects, landmarks, and estimated object locations. In Landmark Promotion SLAM, only reliable mapped landmarks are promoted through various layers of SLAM to generate larger maps.

Claims

1. A method for real-time tracking of features, the method comprising:

acquiring a collection of images and one or more subsequent collections of images to create a three-dimensional space, wherein the collection of images is composed of a first plurality of images captured substantially simultaneously from multiple sources and the one or more subsequent collections of images are composed of a second plurality of images captured substantially simultaneously from the multiple sources, and wherein the second plurality of images are captured later in time than the first plurality of images;

extracting features in each image among the first plurality of images, wherein each feature is a region of multiple pixels within the corresponding image, each region having a feature descriptor vector describing the region;

applying a refinement routine to each feature descriptor vector to identify sub-features indicative of a prominent point within each feature, wherein when a sub-feature cannot be identified within a feature descriptor vector, the corresponding feature is deleted;

identifying a candidate feature in the three-dimensional space for tracking; and

tracking the candidate feature in the three-dimensional space over time using the sub-features within the second plurality of images, wherein the tracked candidate feature is used to determine a travelled path of the multiple sources.

2. The method of claim 1, wherein the multiple sources comprise left and right cameras.

3. The method of claim 2, wherein the collection of images and the one or more subsequent collections of images are captured utilizing stereo imaging by the left and right cameras.

4. The method of claim 3, wherein identifying the candidate feature comprises:

extracting a left feature from an acquired left image captured by the left camera;

extracting a right feature from an acquired right image captured by the right camera, the right image being captured at substantially the same time as the left image;

refining a location of the left feature to a sub-pixel accuracy;

refining a location of the right feature to a sub-pixel accuracy;

matching the left and right features to create a stereo feature;

filtering the stereo feature;

validating the stereo feature; and

returning the validated stereo feature as the candidate feature.

5. The method of claim 4, wherein the extracting of the left and right features utilizes a speeded-up robust features algorithm, and wherein the extracting of the left and right features utilizes a speeded-up robust features algorithm.

6. The method of claim 4, wherein filtering the stereo feature comprises utilizing a matching strength criteria.

7. The method of claim 4, wherein validating the stereo feature comprises utilizing epipolar geometry.

8. The method of claim 4, wherein identifying a candidate feature further comprises creating a list of candidate features by:

extracting a subsequent left feature from a subsequently acquired image by the left camera;

extracting a subsequent right feature from a subsequently acquired right image captured by the right camera;

refining a location of the subsequent left feature to a sub-pixel accuracy;

refining a location of the subsequent right feature to a sub-pixel accuracy;

matching the subsequent left and right features to create a subsequent stereo feature;

filtering the subsequent stereo feature;

validating the subsequent stereo feature;

returning the validated subsequent stereo feature as a subsequent candidate feature; and

adding the subsequent candidate feature to the list of candidate features.

9. The method of claim 3, wherein tracking the candidate feature comprises:

extracting a left feature from an acquired left image captured by the left camera;

extracting a right feature from an acquired right image captured by the right camera, the right image being captured at substantially the same time as the left image;

initializing a left Lucas-Kanade tracker with the left feature;

initializing a right Lucas-Kanade tracker with the right feature;

tracking the left feature on a subsequently acquired left image, the subsequently acquired left image being an image subsequently captured by the left camera;

tracking the right feature on a subsequently acquired right image, the subsequently acquired right image being an image subsequently captured by the right camera;

matching the tracked left and right features to create a tracked stereo feature;

validating the tracked stereo feature; and

returning the validated stereo feature as the tracked candidate feature.

10. The method of claim 9, wherein matching the tracked left and right features comprises utilizing a speeded-up robust features algorithm.

11. The method of claim 9, wherein validating the tracked stereo feature comprises utilizing epipolar geometry.

12. The method of claim 1, wherein identifying the candidate feature comprises selecting features from each image in the collection of images and correlating the features using feature descriptors.

13. The method of claim 1, wherein identifying the candidate feature comprises processing the collection of images through a speeded-up robust features algorithm.

14. The method of claim 13 further comprising validating the candidate feature using epipolar geometry.

15. The method of claim 1, wherein tracking the candidate feature comprises processing the candidate feature through a Lucas-Kanade feature tracker.

16. The method of claim 1, wherein the three dimensional space includes a plurality of corners, further comprising discovering a most prominent corner bounded by the feature by utilizing a sub-pixel corner finding routine.

17. The method of claim 16, wherein the sub-pixel corner finding routine is a Lucas-Kanade optic flow routine.