System and method for hybrid simultaneous localization and mapping of 2D and 3D data acquired by sensors from a 3D scene

• US 9,807,365 B2
• Filed: 12/08/2015
• Issued: 10/31/2017
• Est. Priority Date: 12/08/2015
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Cansizoglu, Esra, Taguchi, Yuichi, Ramalingam, Srikumar
• Primary Examiner(s): BEKELE, MEKONEN T
• Application Number:
  US14/962,257
• Publication Number:
  US 20170161901A1
• Time in Patent Office: 693 Days
• Field of Search: 382154, 382107, 382295, 382103
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/207 using a single 2D image sensor

Abstract

A method and system for registering data by first acquiring the data from a scene by a sensor at different viewpoints, and extracting, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines. A first set of primitives represented in a first coordinate system of the sensor is selected, wherein the first set of primitives includes at least three 3D points. A second set of primitives represented in a second coordinate system is selected, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives. Then, using the first set of primitives and the second set of primitives, the 3D points are registered with each other, and the 3D points with the 3D lines to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system.

Claims

1. A method for registering data, comprising steps of:

acquiring the data from a scene by a sensor at different viewpoints;

extracting, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines, wherein each 3D line is extracted as a ray connecting an optional center of the sensor with a two-dimensional (2) point;

selecting a first set of primitives represented in a first coordinate system of the sensor, wherein the first set of primitives includes at least three 3D points;

selecting a second set of primitives represented in a second coordinate system, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives; and

registering, using the first set of primitives and the second set of primitives, the 3D points with each other using point-to-point correspondence, and the 3D points with the 3D lines using point-to-line correspondence to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system, and wherein the steps are performed in a processor.

2. The method of claim 1, further comprising:

storing the registered primitives in a SLAM map; and

optimizing the SLAM map by bundle adjustment.

3. The method of claim 1, wherein the sensor acquires the data as a sequence of frames for the different viewpoints of the scene, wherein each frame includes a pixel image and a corresponding depth map, and the steps are performed for each frame.

4. The method of claim 3, further comprising:

extracting keypoints from the data, wherein the keypoints include 3D keypoints with valid depths and two-dimensional (2D) keypoints without valid depths;

backprojecting the 3D keypoints with the valid depths to generate the 3D points, and representing the 2D keypoints with the invalid depths as lines passing through the 2D keypoint and an optical center of the sensor based on intrinsic parameters of the sensor to generate the 3D lines.

5. The method of claim 4, wherein the registering further comprising:

determining correspondences between the first set of primitives and the second set of primitives based on matching descriptors to produce point-to-point, point-to-line, and line-to-line correspondences.

6. The method of claim 1, wherein the registering is jointly.

7. The method of claim 1, wherein the data are acquired by multiple sensors.

8. The method of claim 1, wherein the registering of the 3D points uses a perspective-three-point procedure.

9. The method of claim 1, wherein the primitivies include 3D planes.

10. The method of claim 9, wherein registering considers all possible plane-to-plane correspondences.

11. The method of claim 9, wherein registering is performed via random sample consensus (RANSAC), which is initialized with triplets of correspondences in a following order of 3 plane-to-plane correspondences, 2 plane-to-plane and 1 point-to-point correspondences, 1 plane-to-plane and 2 point-to-point correspondences, 3 point-to-point correspondences, and 3 point-to-line correspondences.

12. A system for registering data, comprising:

a sensor, at different viewpoints, configured to acquiring the data from a scene; and

processor configured to extract, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines wherein each 3D line is extracted as a ray connecting an optical center of the sensor with a two-dimensional (2D) point, to select a first set of primitives represented in a first coordinate system of the sensor, wherein the first set of primitives includes at least three 3D points, to select a second set of primitives represented in a second coordinate system, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives, and to register, using the first set of primitives and the second set of primitives, the 3D points with each other using point-to-point correspondence, and the 3D points with the 3D lines using point-to-line correspondence to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system.

13. A method for registering data, comprising steps of:

acquiring the data from a scene by a sensor at different viewpoints;

extracting, from the data, three-dimensional (3D) points and 3D lines, and descriptors associated with the 3D points and the 3D lines;

selecting a first set of primitives represented in a first coordinate system of the sensor, wherein the first set of primitives includes at least three 3D points;

selecting a second set of primitives represented in a second coordinate system, wherein the second set of primitives includes any combination of 3D points and 3D lines to obtain at least three primitives; and

registering, using the first set of primitives and the second set of primitives, the 3D points with each other, and the 3D points with the 3D lines to obtain registered primitives, wherein the registered primitives are used in a simultaneous localization and mapping (SLAM) system, wherein the primitives include 3D planes, wherein registering is performed via random sample consensus (RANSAC), which is initialized with triplets of correspondences in a following order of 3 plane-to-plane correspondences, 2 plane-to-plane and 1 point-to-point correspondences, 1 point-to-plane and 2 point-to-point correspondences, 3 point-to-point correspondences, and 3 point-to-point correspondences, and wherein the steps are performed in a processor.