Method and System for Detecting and Tracking Objects and SLAM with Hierarchical Feature Grouping

• US 20170161546A1
• Filed: 12/08/2015
• Published: 06/08/2017
• Est. Priority Date: 12/08/2015
• Status: Abandoned Application

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Taguchi, Yuichi, Cansizoglu, Esra
• Application Number:
  US14/962,239
• Publication Number:
  US 20170161546A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 10/765 using rules for classification or partitioning the feature space
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0074 Stereoscopic image analysis
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

A method and system detects and localizes an object by first acquiring a frame of a three-dimensional (3D) scene with a sensor, and extracting features from the frame. The frame are segmented into segments, wherein each segment includes one or more features, and for each segment, searching an object map for a similar segment, and only if there is a similar segment in the object map, registering the segment in the frame with the similar segment to obtain a predicted pose of the object. The predicted poses are combined to obtain the pose of the object, which can be outputted.

Claims

1. A method for detecting and localizing an object, comprising steps:

acquiring a frame of a three-dimensional (3D) scene with a sensor;

extracting features from the frame;

segmenting the frame into segments, wherein each segment includes one or more features, and for each segment comprising;

searching an object map for a similar segment, and only if there is a similar segment in the object map, registering the segment in the frame with the similar segment to obtain a predicted pose of the object;

combining the predicted poses to obtain the pose of the object; and

outputting the pose, wherein the steps are performed in a processor.

2. The method of claim 1, wherein the combining further comprises:

refining and merging the predicted poses.

3. The method of claim 2, wherein the refining is a prediction-based registration between the features of the frame and the features of the object map.

4. The method of claim 1, wherein the searching uses a vector of locally aggregated descriptors (VLAD).

5. The method of claim 1, wherein the data are acquired with a depth sensor.

6. The method of claim 1, further comprising:

constructing, with user interaction, the object map offline by scanning known objects.

7. The method of claim 1, wherein the segmenting uses depth-based segmentation.

8. The method of claim 1, wherein the features are associated with descriptors.

9. The method of claim 1, wherein the registering uses random sample consensus (RANSAC).

10. The method of claim 1, further comprising

picking up the object with a robot arm according to the pose.

11. The method of claim 1, wherein the searching is an appearance-based similarity search.

12. A simultaneous localization and mapping (SLAM) method, comprising steps:

determining whether a SLAM map includes any objects, and if no, applying the method of claim 1 to obtain poses of any objects in the frame, and if yes, applying prediction-based object localization to the frame to obtain the poses of the objects;

merging, for each object, similar poses; and

determining if any of the objects are not in the SLAM map, and if no, processing a next frame, and otherwise, if yes, adding the frame, the objects, and the poses to the SLAM map.

13. The method of claim 12, further comprising:

performing bundle adjustment on the SLAM map using constraints to globally optimize the SLAM map.

14. The method of claim 12, wherein the features include 3D points, two-dimensional (2D) points, and 3D planes.

15. A system for detecting, and localizing an object, comprising:

a sensor configured to acquire a frame of a three-dimensional (3D) scene; and

a processor, connected to the sensor, configured to extract features from the frame, to segment the frame into segments, wherein each segment includes one or more features, and for each segment, searching an object map for a similar segment, and only if there is a similar segment in the object map, registering the segment in the frame with the similar segment to obtain a predicted pose of the object, to combine the predicted poses to obtain the pose of the object, and to output the pose.