Methods, systems and apparatus for segmenting objects

• US 10,354,411 B2
• Filed: 12/20/2016
• Issued: 07/16/2019
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Original Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Inventors: Fu, Bo, Zhang, Yan, Gu, Ye, Williams, Jay J., O'Connell, Kevin J.
• Primary Examiner(s): Wang, Carol
• Application Number:
  US15/385,113
• Publication Number:
  US 20180174325A1
• Time in Patent Office: 938 Days
• CPC Class Codes: G01B 11/02 for measuring length, width or thickness G01B11/08 takes precedence
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30112 Baggage; Luggage; Suitcase
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/75 Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

Methods, systems and apparatus for segmenting objects are provided. A controller: receives, from one or more sensors, a 3D representation of a scene that includes an object represented by a 3D model; determines an orientation of a 2D image in the 3D representation by matching the 2D image with a corresponding region of the 3D representation; determines a first portion of the 3D representation that corresponds to a subset of the 3D model of the object from: the orientation of the 2D image in the 3D representation; and predetermined matching data representative of a matching of the 2D image with a respective corresponding region of the 3D model, each of the 3D model, the 2D image, and the predetermined matching data stored in a memory accessible to the controller; segments the first portion of the 3D representation from a second portion of the 3D representation.

Claims

1. A method of dimensioning items, comprising:

receiving, at a controller, from one or more sensors, a three-dimensional (3D) representation of a scene that includes an object represented by a 3D model;

determining, at the controller, an orientation of a 2D image in the 3D representation by matching the 2D image with a corresponding region of the 3D representation;

determining, at the controller, a first portion of the 3D representation that corresponds to a subset of the 3D model of the object from;

the orientation of the 2D image in the 3D representation; and

predetermined matching data representative of a matching of the 2D image with a respective corresponding region of the 3D model, each of the 3D model, the 2D image, and the predetermined matching data stored in a memory accessible to the controller;

segmenting, at the controller, the first portion of the 3D representation from a second portion of the 3D representation, wherein the segmenting the first portion of the 3D representation from the second portion of the 3D representation includes removing the first portion of the 3D representation that corresponds to the subset of the 3D model; and

dimensioning one or more items represented by the second portion of the 3D representation that remains after the first portion of the 3D representation that corresponds to the 3D model is removed.

2. The method of claim 1, wherein the 3D representation of the scene comprises one or more of a point cloud, color data associated with the point cloud, and a color point cloud.

3. The method of claim 1, wherein each of the determining the orientation of the 2D image in the 3D representation, and the determining the first portion of the 3D representation that corresponds to a subset of the 3D model of the object includes 3D point cloud feature matching and color feature matching.

4. The method of claim 1, wherein the 3D representation of the scene comprises a plurality of images from the one or more sensors, and the matching of the 2D image with a corresponding region of the 3D representation comprises performing a 2D matching between the 2D image and one image of the plurality of images.

5. The method of claim 4, further comprising determining a direction of travel of the object in the scene, and determining the one image of the plurality of images to use in the 2D matching from the direction of travel.

6. The method of claim 1, further comprising optimizing the 3D representation by one or more of:

a 3D Hough transformation, geometry consistence clustering, iterative closest point registration refinement and 3D hypothesis verification.

7. The method of claim 1, wherein the segmenting the first portion of the 3D representation from the second portion includes determining a bounding box of the subset of the 3D model in the 3D representation and removing data corresponding to the bounding box from the 3D representation.

8. The method of claim 1, wherein the 2D image comprises a representation of markings on the object, the markings including one or more of a logo, a given portion of the object, a feature-rich image, and a unique portion of the object.

9. The method of claim 1, wherein the 3D model of the object comprises a 3D model of a forklift, and the subset of the 3D model excludes forks of the forklift.

10. A device for dimensioning items, comprising:

a controller, and a communication interface, the controller configured to;

receive, using the communication interface, from one or more sensors, a three-dimensional (3D) representation of a scene that includes an object represented by a 3D model;

determine an orientation of a 2D image in the 3D representation by matching the 2D image with a corresponding region of the 3D representation;

determine a first portion of the 3D representation that corresponds to a subset of the 3D model of the object from;

the orientation of the 2D image in the 3D representation; and

predetermined matching data representative of a matching of the 2D image with a respective corresponding region of the 3D model, each of the 3D model, the 2D image, and the predetermined matching data stored in a memory accessible to the controller;

segment the first portion of the 3D representation from a second portion of the 3D representation by removing the first portion of the 3D representation that corresponds to the subset of the 3D model; and

dimension one or more items represented by the second portion of the 3D representation that remains after the first portion of the 3D representation that corresponds to the 3D model is removed.

11. The device of claim 10, wherein each of determining the orientation of the 2D image in the 3D representation, and determining the first portion of the 3D representation that corresponds to a subset of the 3D model of the object includes:

3D point cloud feature matching and color feature matching.

12. The device of claim 10, wherein the 3D representation of the scene comprises a plurality of images from the one or more sensors, and the controller is further configured to match the 2D image with a corresponding region of the 3D representation by performing a 2D matching between the 2D image and one image of the plurality of images.

13. The device of claim 12, the controller is further configured to determine a direction of travel of the object in the scene, and determine the one image of the plurality of images to use in the 2D matching from the direction of travel.

14. The device of claim 10, the controller is further configured to optimize the 3D representation by one or more of:

a 3D Hough transformation, geometry consistence clustering, iterative closest point registration refinement and 3D hypothesis verification.

15. The device of claim 10, the controller is further configured to segment the first portion of the 3D representation from the second portion by determining a bounding box of the subset of the 3D model in the 3D representation and removing data corresponding to the bounding box from the 3D representation.

16. A non-transitory computer-readable medium storing a computer program, wherein execution of the computer program causes a machine to perform operations comprising:

receiving, at a controller, from one or more sensors, a three-dimensional (3D) representation of a scene that includes an object represented by a 3D model;

determining, at the controller, an orientation of a 2D image in the 3D representation by matching the 2D image with a corresponding region of the 3D representation;

determining, at the controller, a first portion of the 3D representation that corresponds to a subset of the 3D model of the object from;

the orientation of the 2D image in the 3D representation; and

predetermined matching data representative of a matching of the 2D image with a respective corresponding region of the 3D model, each of the 3D model, the 2D image, and the predetermined matching data stored in a memory accessible to the controller;

segmenting, at the controller, the first portion of the 3D representation from a second portion of the 3D representation by removing the first portion of the 3D representation that corresponds to the subset of the 3D model; and

dimensioning one or more items represented by the second portion of the 3D representation that remains after the first portion of the 3D representation that corresponds to the 3D model is removed.