ROBOTIC SYSTEM WITH AUTOMATED PACKAGE REGISTRATION MECHANISM AND AUTO-DETECTION PIPELINE

• US 20200130962A1
• Filed: 10/29/2019
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: MUJIN Inc.
• Original Assignee: MUJIN Inc.
• Inventors: Yu, Jinze, Rodrigues, Jose Jeronimo Moreira, Diankov, Rosen Nikolaev
• Granted Patent: US 11,189,033 B2
• Time in Patent Office: Days
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65B 43/46 using grippers
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: G05B 19/40935 Selection of predetermined shapes and defining the dimensions with parameter input
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: G05B 2219/40607 Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06T 7/50 : Depth or shape recovery
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation

Abstract

The present disclosure relates to detecting and registering unrecognized or unregistered objects. A minimum viable range (MVR) may be derived based on inspecting image data that represents objects at a start location. The MVR may be determined to be a certain MVR or an uncertain MVR according to one or more features represented in the image data. The MVR may be used to register corresponding objects according to the certain or uncertain determination.

Claims

1. A method of operating a robotic system, the method comprising:

receiving two-dimensional (2D) image data and three-dimensional (3D) image data representative of a target stack including one or more unrecognized objects;

identifying an unexplained region of the 2D and/or the 3D image data based on a comparison with an object set representative of one or more physical aspects of registered objects;

identifying one or more layers within the unexplained region of the 2D image data and/or the 3D image data, wherein each layer includes a set of laterally adjacent points having depth measures within a threshold range from each other in the 3D image data;

identifying exposed edges in the unexplained region, wherein the exposed edges represent boundaries of the one or more layers where the depth measures deviate outside the threshold range across horizontally adjacent locations;

identifying one or more exposed corners based on the exposed edges, wherein each of the exposed corners correspond to a junction between a set of the exposed edges;

deriving a minimum viable region (MVR) based on the identified exposed edges and the exposed corners, wherein the MVR represents an estimate of a surface for an unregistered object represented in the unexplained region;

determining a number of the exposed corners associated with the derived MVR;

detecting an unrecognized object according to the MVR and the number of exposed corners; and

registering the unrecognized object based on updating the object set to include the MVR and/or a processing result thereof as representations of the unrecognized object.

2. The method of claim 1, further comprising iteratively updating and analyzing the unexplained region according to the updated object set based on:

comparing the unexplained region with the updated object set;

identifying a portion within the unexplained region that match the MVR and/or the processing result thereof used to update the object set;

detecting an object according to the portion that match the MVR and/or the processing result thereof; and

updating the unexplained region by removing the portion that match the MVR and/or the processing result thereof.

3. The method of claim 2, wherein:

the object set is an empty set for an initial iteration;

the unexplained region includes the 2D image data, the 3D image data, a portion thereof, or a combination thereof for the initial iteration; and

registering the unrecognized object includes autonomously populating the object set while implementing tasks involving the unrecognized objects located in the target stack.

4. The method of claim 1, wherein:

detecting the unrecognized object includes detecting the unrecognized object directly when the number of the exposed corners exceed a predetermined threshold; and

registering the recognized object includes storing the MVR in the object set directly when the exposed corners exceed the predetermined threshold.

5. The method of claim 4, wherein the predetermined threshold is two.

6. A robotic system, comprising:

at least one processor; and

at least one memory device connected to the at least one processor and having stored thereon instructions executable by the processor for;

receiving one or more image data representative of one or more unrecognized objects at a start location;

deriving a minimum viable region (MVR) based on identifying one or more exposed corners in the one or more image data, wherein each of the exposed corners represent a junction between exposed edges that are not horizontally adjacent to another surface or object;

updating an object set with the MVR and/or a processing result thereof based on a number of the exposed corners associated with the MVR, wherein;

the object set represents one or more physical aspects of previously registered objects, and

the MVR and/or the processing result thereof represent one or more physical attributes of one of the unrecognized objects at the start location; and

recognizing another object at the start location based on the updated object set.

7. The robotic system of claim 6, wherein the at least one memory device includes instructions for determining the MVR as a certain MVR based on comparing the number of the exposed corners to a predetermined threshold.

8. The robotic system of claim 7, wherein the at least one memory device includes instructions for determining the MVR as the certain MVR when the MVR is associated with three or more of the exposed corners.

9. The robotic system of claim 8, wherein the at least one memory device includes instructions for updating the object set with the MVR and/or a processing result thereof directly based on the determination of the MVR as the certain MVR and without further updating or adjusting the MVR.

10. The robotic system of claim 9, wherein the at least one memory device includes instructions for:

identifying an unexplained region of the image data that does not return a match with objects represented in the object set;

deriving the MVR based on the unexplained region, wherein the certain MVR is within the unexplained region; and

removing a portion of the unexplained region that matches the certain MVR for further object recognition and/or manipulation.

11. The robotic system of claim 10, wherein the at least one memory device includes instructions for:

implementing operations for transferring an unrecognized object that corresponds to the certain MVR away from the start location;

deriving a subsequent MVR based on an updated unexplained region after implementing the operations for transferring the unrecognized object, wherein the subsequent MVR represents a further unrecognized object at the start location and is represented in the updated unexplained region; and

updating the object set with the second MVR and/or a processing result thereof.

12. The robotic system of claim 6, wherein the at least one memory device includes instructions for:

determining the MVR as an uncertain MVR when the MVR is associated with two or fewer of the exposed corners; and

updating the object set with the MVR and/or a processing result thereof based on further analyzing the uncertain MVR and/or manipulating one or more of the unrecognized objects at the start location.

13. The robotic system of claim 6, wherein the at least one memory device includes instructions for:

detecting one or more recognized objects based on comparing the image data with the object set;

identifying an unexplained region of the image data that does not return a match with the object set; and

deriving the MVR within the unexplained region based on identifying the one or more exposed corners in the unexplained region of the image data, wherein the MVR represents an estimate of a surface of the one of the unrecognized objects at the start location.

14. The robotic system of claim 13, wherein the at least one memory device includes instructions for:

implementing operations for transferring the one or more recognized objects away from the start location;

updating the MVR after implementing the operation to transfer the one or more recognized objects; and

updating the object set with the updated MVR and/or a processing result thereof.

15. A tangible, non-transient computer-readable medium having processor instructions stored thereon that, when executed by a processor, cause the processor to implement a method, the method comprising:

iteratively analyzing one or more received image data that represent registered and/or unrecognized objects at a start location by;

deriving one or more minimum viable regions (MVRs) based on identifying one or more exposed corners in the received image data, wherein each of the exposed corners represent a junction between exposed edges that are not horizontally adjacent to another surface or object;

determining a classification for each of the MVRs based on a number of the exposed corners associated with the corresponding MVR;

updating an object set with the MVRs and/or a processing result thereof according to the corresponding classification, wherein;

the object set represents one or more physical aspects of previously registered objects, and

the each of the MVRs and/or the processing result thereof represent one or more physical attributes of one of the unrecognized objects at the start location; and

recognizing other objects at the start location using the updated object set.

16. The non-transient computer-readable medium of claim 15, wherein:

the one or more received image data include two-dimensional (2D) image data and three-dimensional (3D) image data both representing the target stack from one imaging location;

the method further comprises;

generating a detection estimate based on comparing the 2D image data to the object set; and

verifying the detection estimate based on the 3D image data.

17. The non-transient computer-readable medium of claim 16, wherein:

generating the detection estimate includes generating the detection estimate based matching a portion of the 2D image data to a surface image of an object represented in the object set;

verifying the detection estimate includes verifying the detection estimate based on;

identifying a portion in the 3D image data corresponding to the detection estimate;

calculating lengths for 3D edges in the identified portion of the 3D image data; and

verifying the detection estimate when the lengths of the 3D edges match those of the object represented in the object set.

18. The non-transient computer-readable medium of claim 15, wherein:

the one or more received image data include two-dimensional (2D) image data and three-dimensional (3D) image data both representing the target stack from one imaging location;

the method further comprises;

generating a detection estimate based on comparing the 3D image data to the object set; and

verifying the detection estimate based on the 2D image data.

19. The non-transient computer-readable medium of claim 18, wherein:

generating the detection estimate includes generating the detection estimate based on;

identifying a layer within the 3D image data, wherein the layer includes a set of horizontally adjacent locations having depth measures within a threshold range of each other,

identifying edges of the layer, wherein the edges represent locations where the depth measures deviate outside of the threshold range across horizontally adjacent locations,

calculating lengths of the edges, and

matching the edge lengths to dimensions of an object represented in the object set; and

verifying the detection estimate includes verifying the detection estimate based on;

identifying a portion in the 2D image data corresponding to the detection estimate,

identifying 2D characteristics in the identified portion of the 2D image data, and

verifying the detection estimate when the 2D characteristics match corresponding characteristics of an image of the one of the registered objects represented in the object set.

20. The non-transient computer-readable medium of claim 15, wherein the method further comprises iteratively building the object set from an empty set based on analyzing the one or more received image data by:

identifying the one or more received image data or a portion thereof as an unexplained region when the object set includes no registration data of any objects;

deriving the one or more MVRs from the unexplained region;

storing at least a subset of the one or more MVRs in the object set according to the corresponding classifications;

detecting a first set of objects represented in the received image data based on matching portions of the received image data to the one or more MVRs stored in the object set;

updating the unexplained region by removing the matched portions; and

deriving a new set of MVRs based on the updated unexplained region for registering a second set of objects represented in the received image data.