SPARSE SIMULTANEOUS LOCALIZATION AND MATCHING WITH UNIFIED TRACKING

• US 20180005015A1
• Filed: 06/29/2017
• Published: 01/04/2018
• Est. Priority Date: 07/01/2016
• Status: Abandoned Application

• Current Assignee: VanGogh Imaging, Inc.
• Original Assignee: VanGogh Imaging, Inc.
• Inventors: Hou, Xin, Cambias, Craig
• Application Number:
  US15/638,278
• Publication Number:
  US 20180005015A1
• Time in Patent Office: Days
• CPC Class Codes: G01S 13/06 Systems determining position data of a target
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 17/02 Systems using the reflection of electromagnetic waves other than radio waves G01S17/66 takes precedence
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Described herein are methods and systems for tracking a pose of one or more objects represented in a scene. A sensor captures a plurality of scans of objects in a scene, each scan comprising a color and depth frame. A computing device receives a first one of the scans, determines two-dimensional feature points of the objects using the color and depth frame, and retrieves a key frame from a database that stores key frames of the objects in the scene, each key frame comprising map points. The computing device matches the 2D feature points with the map points, and generates a current pose of the objects in the color and depth frame using the matched 2D feature points. The computing device inserts the color and depth frame into the database as a new key frame, and tracks the pose of the objects in the scene across the scans.

Claims

1. A system for tracking a pose of one or more objects represented in a scene, the system comprising:

a sensor that captures a plurality of scans of one or more objects in a scene, each scan comprising a color and depth frame;

a database that stores one or more key frames of the one or more objects in the scene, each key frame comprising a plurality of map points associated with the one or more objects;

a computing device that;

a) receives a first one of the plurality of scans from the sensor;

b) determines two-dimensional (2D) feature points of the one or more objects using the color and depth frame of the received scan;

c) retrieves a key frame from the database;

d) matches one or more of the 2D feature points with one or more of the map points in the key frame;

e) generate a current pose of the one or more objects in the color and depth frame using the matched 2D feature points;

f) insert the color and depth frame into the database as a new key frame, including the matched 2D feature points as map points for the new key frame; and

g) repeat steps a)-f) on each of the remaining scans, using the inserted new key frame for matching in step d);

wherein the computing device tracks the pose of the one or more objects in the scene across the plurality of scans.

2. The system of claim 1, further comprising generating a 3D model of the one or more objects in the scene using the tracked pose information.

3. The system of claim 1, wherein the step of inserting the color and depth frame into the database as a new key frame comprises:

converting the color and depth frame into a new key frame and converting the 2D feature points of the color and depth frame into map points of the new key frame;

fusing one or more map points of the new key frame that have valid depth information with similar map points of one or more neighbor key frames;

estimating a 3D position of one or more map points of the new key frame that do not have valid depth information;

refining the pose of the new key frame and the one or more neighbor key frames fused with the new key frame; and

storing the new key frame and associated map points into the database.

4. The system of claim 3, wherein converting the color and depth frame into a new key frame and converting the 2D feature points of the color and depth frame into map points of the new key frame comprises converting a 3D position of the one or more map points of the new key frame from a local coordinate system to a global coordinate system using the pose of the new key frame.

5. The system of claim 3, wherein the computing device correlates the new key frame with the one or more neighbor key frames based upon a number of map points shared between the new key frame and the one or more neighbor key frames.

6. The system of claim 3, wherein the step of fusing one or more map points of the new key frame that have valid depth information with similar map points of one or more neighbor key frames comprises:

projecting each map point from the one or more neighbor key frames to the new key frame;

identifying a map point with similar 2D features that is closest to a position of the projected map point; and

fusing the projected map point from the one or more neighbor key frames to the identified map point in the new key frame.

7. The system of claim 3, wherein the step of estimating a 3D position of one or more map points of the new key frame that do not have valid depth information comprises:

matching a map point of the new key frame that do not have valid depth information with a map point in each of two neighbor key frames; and

determining a 3D position of the map point of the new key frame using linear triangulation with the 3D position of the map points in the two neighbor key frames.

8. The system of claim 3, wherein the step of refining the pose of the new key frame and the one or more neighbor key frames fused with the new key frame is performed using local bundle adjustment.

9. The system of claim 3, wherein the computing device deletes redundant key frames and associated map points from the database.

10. The system of claim 1, wherein the computing device:

determines a similarity between the new key frame and one or more key frames stored in the database;

estimates a 3D rigid transformation between the new key frame and the one or more key frames stored in the database;

selects a key frame from the one or more key frames stored in the database based upon the 3D rigid transformation; and

merges the new key frame with the selected key frame to minimize drifting error.

11. The system of claim 10, wherein the step of determining a similarity between the new key frame and one or more key frames stored in the database comprises determining a number of matched features between the new key frame and one or more key frames stored in the database.

12. The system of claim 10, wherein the step of estimating a 3D rigid transformation between the new key frame and the one or more key frames stored in the database comprises:

selecting one or more pairs of matching features between the new key frame and the one or more key frames stored in the database;

determining a rotation and translation of each of the one or more pairs; and

selecting a pair of the one or more pairs with a maximum inlier ratio using the rotation and translation.

13. The system of claim 10, wherein the step of merging the new key frame with the selected key frame to minimize drifting error comprises:

merging one or more feature points in the new key frame with one or more feature points in the selected key frame; and

connecting the new key frame to the selected key frame using the merged feature points.

14. A computerized method of tracking a pose of one or more objects represented in a scene, the method comprising:

a) capturing, by a sensor, a plurality of scans of one or more objects in a scene, each scan comprising a color and depth frame;

b) receiving, by a computing device, a first one of the plurality of scans from the sensor;

c) determining, by the computing device, two-dimensional (2D) feature points of the one or more objects using the color and depth frame of the received scan;

d) retrieving, by the computing device, a key frame from a database that stores one or more key frames of the one or more objects in the scene, each key frame comprising a plurality of map points associated with the one or more objects;

e) matching, by the computing device, one or more of the 2D feature points with one or more of the map points in the key frame;

f) generating, by the computing device, a current pose of the one or more objects in the color and depth frame using the matched 2D feature points;

g) inserting, by the computing device, the color and depth frame into the database as a new key frame, including the matched 2D feature points as map points for the new key frame; and

h) repeating, by the computing device, steps b)-g) on each of the remaining scans, using the inserted new key frame for matching in step e);

wherein the server computing device tracks the pose of the one or more objects in the scene across the plurality of scans.

15. The method of claim 14, further comprising generating, by the computing device, a 3D model of the one or more objects in the scene using the tracked pose information.

16. The method of claim 14, wherein the step of inserting the color and depth frame into the database as a new key frame comprises:

converting the color and depth frame into a new key frame and converting the 2D feature points of the color and depth frame into map points of the new key frame;

fusing one or more map points of the new key frame that have valid depth information with similar map points of one or more neighbor key frames;

estimating a 3D position of one or more map points of the new key frame that do not have valid depth information;

refining the pose of the new key frame and the one or more neighbor key frames fused with the new key frame; and

storing the new key frame and associated map points into the database.

17. The method of claim 16, wherein converting the color and depth frame into a new key frame and converting the 2D feature points of the color and depth frame into map points of the new key frame comprises converting a 3D position of the one or more map points of the new key frame from a local coordinate system to a global coordinate system using the pose of the new key frame.

18. The method of claim 16, further comprising correlating the new key frame with the one or more neighbor key frames based upon a number of map points shared between the new key frame and the one or more neighbor key frames.

19. The method of claim 16, wherein the step of fusing one or more map points of the new key frame that have valid depth information with similar map points of one or more neighbor key frames comprises:

projecting each map point from the one or more neighbor key frames to the new key frame;

identifying a map point with similar 2D features that is closest to a position of the projected map point; and

fusing the projected map point from the one or more neighbor key frames to the identified map point in the new key frame.

20. The method of claim 16, wherein the step of estimating a 3D position of one or more map points of the new key frame that do not have valid depth information comprises:

matching a map point of the new key frame that do not have valid depth information with a map point in each of two neighbor key frames; and

determining a 3D position of the map point of the new key frame using linear triangulation with the 3D position of the map points in the two neighbor key frames.

21. The method of claim 16, wherein the step of refining the pose of the new key frame and the one or more neighbor key frames fused with the new key frame is performed using local bundle adjustment.

22. The method of claim 16, further comprising deleting redundant key frames and associated map points from the database.

23. The method of claim 14, further comprising:

determining a similarity between the new key frame and one or more key frames stored in the database;

estimating a 3D rigid transformation between the new key frame and the one or more key frames stored in the database;

selecting a key frame from the one or more key frames stored in the database based upon the 3D rigid transformation; and

merging the new key frame with the selected key frame to minimize drifting error.

24. The method of claim 23, wherein the step of determining a similarity between the new key frame and one or more key frames stored in the database comprises determining a number of matched features between the new key frame and one or more key frames stored in the database.

25. The method of claim 23, wherein the step of estimating a 3D rigid transformation between the new key frame and the one or more key frames stored in the database comprises:

selecting one or more pairs of matching features between the new key frame and the one or more key frames stored in the database;

determining a rotation and translation of each of the one or more pairs; and

selecting a pair of the one or more pairs with a maximum inlier ratio using the rotation and translation.

26. The method of claim 23, wherein the step of merging the new key frame with the selected key frame to minimize drifting error comprises:

merging one or more feature points in the new key frame with one or more feature points in the selected key frame; and

connecting the new key frame to the selected key frame using the merged feature points.