System and method for monocular simultaneous localization and mapping

• US 10,636,198 B2
• Filed: 12/28/2017
• Issued: 04/28/2020
• Est. Priority Date: 12/28/2017
• Status: Active Grant

• Current Assignee: Beijing Jingdong Shangke Information Technology Co., Ltd. (JD.com, Inc.), JD.com American Technologies Corporation (JD.com, Inc.)
• Original Assignee: Beijing Jingdong Shangke Information Technology Co., Ltd. (JD.com, Inc.)
• Inventors: Xu, Yi, Yang, Shanglin, Wu, Yuzhang, Zhou, Hui
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/857,501
• Publication Number:
  US 20190206116A1
• Time in Patent Office: 852 Days
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/75 involving models

Abstract

A method for simultaneous localization and mapping. The method includes the step of detecting two-dimensional (2D) feature points from a current frame captured by a camera; matching the 2D feature points from the current frame directly to three-dimensional (3D) map points in a 3D map, so as to obtain correspondence between the 2D feature points and the 3D map points; and computing a current pose of the camera based on the obtained correspondence. Each of the 2D feature points and the 3D map points has a feature descriptor. The step of matching is performed by comparing the feature descriptors of the 2D feature points and the feature descriptors of the 3D map points.

Claims

1. A method for simultaneous localization and mapping, comprising:

detecting two-dimensional (2D) feature points from a current frame captured by a camera;

matching the 2D feature points from the current frame directly to three-dimensional (3D) map points in a 3D map independently of correspondence between the 3D map points and 2D feature points from any frames previous to the current frame, so as to obtain correspondence between the 2D feature points from the current frame and the 3D map points;

computing a current pose of the camera based on the obtained correspondence; and

in response to determining that the current frame qualifies as a key frame;

reconstructing new 3D map points based on the current frame, and incorporating the new 3D map points to the 3D map,

wherein each of the 2D feature points and the 3D map points has a feature descriptor, and the matching is performed by comparing the feature descriptors of the 2D feature points and the feature descriptors of the 3D map points.

2. The method according to claim 1, wherein the current frame is determined to qualify as a key frame when a percentage of 2D feature points of the current frame that have been matched to the 3D map points is below a predetermined threshold.

3. The method according to claim 2, wherein the predetermined threshold is a value between 5% and 50%.

4. The method according to claim 1, further comprising, in response to determining that the current frame qualifies as a key frame:

respectively increasing ages of the 3D map points in the updated 3D map by 1.

5. The method according to claim 4, wherein the matching is performed on at least a part of the 3D map points whose ages are less than a predetermined threshold.

6. The method according to claim 4, further comprising:

removing 3D map points whose ages are more than a predetermined threshold.

7. The method according to claim 1, further comprising:

rendering a virtual object on the 3D map based on the current frame.

8. The method according to claim 1, further comprising:

selecting a pair of frames from a sequence of frames captured by the camera based on predetermined criteria; and

constructing an initial 3D map by;

matching 2D feature points from the pair of frames to obtain pairs of 2D feature points, recovering relative rotation and translation of the camera based on the pairs of 2D feature points, and triangulating the pair of 2D feature points to obtain the initial 3D map points.

9. The method according to claim 1, further comprising computing the current pose of the camera based on data obtained from at least one of:

an accelerometer;

a gyroscope; and

an inertial measurement unit (IMU).

10. The method according to claim 1, further comprising optimizing the current pose of the camera by bundle adjustment.

11. The method according to claim 1, wherein each of the feature descriptors of the 2D feature points comprises a vector or a histogram of a corresponding one of the 2D feature points, and each of the feature descriptors of the 3D map points comprises a vector or a histogram of a corresponding one of the 3D map points.

18. A non-transitory tangible computer readable medium which stores computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to perform the method according to claim 1.

12. A device for simultaneous localization and mapping, comprising a camera, at least one processor, and a memory including instructions which, when executed by the at least one processor, cause the device to:

detect two-dimensional (2D) feature points from a current frame captured by the camera;

match the 2D feature points from the current frame directly to three-dimensional (3D) map points in a 3D map independently of correspondence between the 3D map points and 2D feature points from any frames previous to the current frame, so as to obtain correspondence between the 2D feature points from the current frame and the 3D map points;

compute a current pose of the camera based on the obtained correspondence; and

in response to determining that the current frame qualifies as a key frame;

reconstruct new 3D map points based on the current frame, and incorporate the new 3D map points to the 3D map,

wherein each of the 2D feature points and the 3D map points has a feature descriptor, and the matching is performed by comparing the feature descriptors of the 2D feature points and the feature descriptors of the 3D map points.

13. The device according to claim 12, wherein the instructions which, when executed by the at least one processor, further cause the device to determine that the current frame qualifies as a key frame when a percentage of 2D feature points of the current frame that have been matched to the 3D map points is below a predetermined threshold having a value between 5% and 50%.

14. The device according to claim 12, wherein the instructions which, when executed by the at least one processor, further cause the device to, in response to determining that the current frame qualifies as a key frame:

respectively increase ages of the 3D map points in the updated 3D map by 1; and

remove 3D map points whose ages are more than a predetermined threshold.

15. The device according to claim 12, wherein the instructions which, when executed by the at least one processor, further cause the device to:

select a pair of frames from a sequence of frames captured by the camera based on predetermined criteria; and

construct an initial 3D map by;

matching 2D feature points from the pair of frames to obtain pairs of 2D feature points, recovering relative rotation and translation of the camera based on the pairs of 2D feature points, and triangulating the pair of 2D feature points to obtain initial 3D map points.

16. The device according to claim 12, wherein each of the feature descriptors of the 2D feature points comprises a vector or a histogram of a corresponding one of the 2D feature points, and each of the feature descriptors of the 3D map points comprises a vector or a histogram of a corresponding one of the 3D map points.

17. The device according to claim 12, wherein the instructions which, when executed by the at least one processor, further cause the device to:

optimize the current pose of the camera by bundle adjustment.