ROBOTIC SYSTEM WITH AUTOMATED PACKAGE SCAN AND REGISTRATION MECHANISM AND METHODS OF OPERATING THE SAME

• US 20200130963A1
• Filed: 10/29/2019
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: MUJIN Inc.
• Original Assignee: MUJIN Inc.
• Inventors: Diankov, Rosen Nikolaev, Liu, Huan, Ye, Xutao, Rodrigues, Jose Jeronimo Moreira, Kanemoto, Yoshiki, Yu, Jinze, Islam, Russell
• Granted Patent: US 11,501,445 B2
• Time in Patent Office: Days
• CPC Class Codes: B25J 9/1676 Avoiding collision or forbidden zones
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65B 43/46 using grippers
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: G05B 19/40935 Selection of predetermined shapes and defining the dimensions with parameter input
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: G05B 2219/40607 Fixed camera to observe workspace, object, workpiece, global
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/174 : involving the use of two or more images
• CPC Class Codes: G06T 7/50 : Depth or shape recovery
• CPC Class Codes: G06T 7/62 : of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/44 : Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation

Abstract

A system and method for operating a robotic system to scan and register unrecognized objects is disclosed. The robotic system may use an image data representative of an unrecognized object located at a start location to implement operations for transferring the unrecognized object from the start location. While implementing the operations, the robotic system may obtain additional data, including scanning results of one or more portions of the unrecognized object not included in the image data. The robotic system may use the additional data to register the unrecognized object.

Claims

1. A method for operating a robotic system, the method comprising:

receiving an image data representative of a top surface of an unrecognized object located at a start location;

identifying a first set of peripheral edges of the top surface represented in the image data;

implementing operations for transferring the unrecognized object to a task location based at least partially on the first set of peripheral edges;

detecting an event while implementing the operations for transferring the unrecognized object, wherein the event represents an entry or an exit of the unrecognized object relative to a laterally-oriented crossing reference located at a crossing reference height;

calculating an object height of the unrecognized object based on the event;

deriving a location of a vertically-oriented surface or a portion thereof of the unrecognized object based on the object height and the first set of peripheral edges;

deriving a scanning position based on the derived location of the vertically-oriented surface, wherein the scanning position represents a pose for presenting the vertically-oriented surface to a scanning sensor;

implementing operations for transferring the unrecognized object to expose the vertically-oriented surface to a scanning sensor based on the scanning position;

generating a scanning result representative of an output from the scanning sensor scanning the vertically-oriented surface or the portion thereof; and

creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

2. The method of claim 1, further comprising:

generating one or more commands for operating the scanning sensor based on implementing the operations to transfer the unrecognized object to expose the vertically-oriented surface to the scanning sensor; and

wherein;

the scanning result is a value represented by a bar code and/or a Quick Response (QR) code on the vertically-oriented surface.

3. The method of claim 1, further comprising:

deriving a scanning maneuver based on the scanning position, wherein the scanning maneuver is for transferring the unrecognized object to present multiple portions thereof to the scanning sensor;

after implementing operations to transfer the vertically-oriented surface, implementing operations according to the scanning maneuver for presenting multiple portions of the vertically-oriented surface and/or multiple surfaces of the unrecognized object to the scanning sensor; and

generating one or more commands for operating the scanning sensor based on implementing the operations according to the scanning maneuver to scan the multiple portions of the vertically-oriented surface and/or the multiple surfaces.

4. The method of claim 3, wherein the scanning maneuver corresponds to rotating the unrecognized object about a vertical axis.

5. The method of claim 3, further comprising:

calculating a dimension of the vertically-oriented surface based on the object height and/or the first set of peripheral edges;

deriving a second scanning position based on the derived location of the vertically-oriented surface when the dimension exceeds a threshold, wherein the scanning position and the second scanning position are for presenting different end portions or corners of the vertically-oriented surface to the scanning sensor;

wherein;

the scanning maneuver is for transferring the unrecognized object from the scanning position to the second scanning position; and

the one or more commands for operating the scanning sensor represents keeping the scanning sensor active while implementing operations according to the scanning maneuver.

6. A robotic system comprising:

at least one processor; and

at least one memory device connected to the at least one processor and having stored thereon instructions executable by the processor for;

receiving an image data representative of an unrecognized object located at a start location;

implementing a task based on the image data, wherein the task includes operations for transferring the unrecognized object to a task location;

while implementing the operations for transferring the unrecognized object, implement operations for scanning at least one portion of the unrecognized object that was not represented in the image data;

generating a scanning result based on implementing the operations for scanning the at least one portion of the unrecognized object; and

creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

7. The robotic system of claim 6, wherein the at least one memory device includes instructions for:

obtaining additional data corresponding to the unrecognized object while implementing the task;

calculating an object height of the unrecognized object based on the additional data;

deriving a scanning position based on the object height, wherein the scanning position represents a pose for presenting to a scanning sensor the at least one portion of the unrecognized object that was not represented in the image data;

implementing operations for scanning at least one portion of the unrecognized object based on;

a command and/or a setting for positioning the unrecognized object according to the scanning position; and

an activation command for operating the scanning sensor while and/or after positioning the unrecognized object according to the scanning position.

8. The robotic system of claim 7, wherein:

the image data represents a top surface of the unrecognized object;

the at least one memory device includes instructions for;

identifying locations of one or more peripheral edges of the top surface represented in the image data;

deriving a target location of the unrecognized object based on the one or more peripheral edges and the object height, wherein the target location represents a position of the at least one portion of the unrecognized object that was not represented in the image data;

determining a scanning zone of the scanning sensor, wherein the scanning zone represents a space detectable by the scanning sensor; and

deriving the scanning position based on the target location and the scanning zone.

9. The robotic system of claim 7, wherein the at least one portion of the unrecognized object that was not represented in the image data includes a vertically-oriented surface or a peripheral portion thereof of the unrecognized object.

10. The robotic system of claim 9, wherein:

the scanning position corresponds to a first corner of the vertically-oriented surface; and

the at least one memory device includes instructions for implementing operations for scanning based on;

a second scanning position derived based on the object height, wherein the second scanning position represents a pose for presenting to the scanning sensor a second corner of the vertically-oriented surface; and

a scanning maneuver derived according to the scanning position and the second scanning position for vertically and/or horizontally moving the vertically-oriented surface.

11. The robotic system of claim 10, wherein the at least one memory device includes instructions for generating the scanning result including an identifier location based on:

receiving an identifier value resulting from scanning an identifier accessible on the vertically-oriented surface;

determining a gripper location at a time of receiving the identifier value;

deriving an identifier location based on the gripper location, wherein the identifier location represents a position of the identifier on the vertically-oriented surface; and

generating the scanning result including the identifier value and the identifier location.

12. The robotic system of claim 7, wherein the at least one memory device includes instructions for implementing operations for scanning at least one portion of the unrecognized object based on further generating a command and/or a setting for rotating the unrecognized object about a vertical axis to present multiple surfaces to the scanning sensor.

13. The robotic system of claim 6, wherein the at least one memory device includes instructions for:

receiving a scanned image, wherein the scanned image is a visual representation of the at least one portion of the unrecognized object; and

generating the scanning result based on the scanning image.

14. The robotic system of claim 13, wherein the at least one memory device includes instructions for:

determining an identifier value based on visually processing the scanned image, wherein the identifier value corresponds to an identifier accessible on the vertically-oriented surface;

deriving an identifier location based on visually processing the scanned image, wherein the identifier location represents a position of the identifier on the vertically-oriented surface; and

generating the scanning result including the identifier value and the identifier location.

15. The robotic system of claim 13, wherein the at least one memory device includes instructions for:

identifying peripheral edges of the unrecognized object in the scanned image; and

generating the scanning result including a cropped portion of the scanned image within the identified peripheral edges.

16. A tangible, non-transient computer-readable medium having processor instructions stored thereon that, when executed by a robotic system via one or more processors thereof, cause the robotic system to perform a method, the method comprising:

receiving an image data representative of an unrecognized object located at a start location;

locating at least a portion of the unrecognized object based on the image data;

implementing a task based at least partially on the located portion, wherein the task is for transferring the unrecognized object to a task location;

obtaining additional information while implementing the task;

deriving one or more scanning positions, based on the obtained additional information, for presenting to a scanning sensor at least one portion of the unrecognized object that was not represented in the image data;

while implementing the task, implementing operations, based on the one or more scanning positions, for scanning the at least one portion of the unrecognized object that was not represented in the image data;

generating a scanning result based on implementing the operations for scanning the at least one portion of the unrecognized object; and

creating new registration data representative of a new record for the unrecognized object, wherein the new registration data includes the scanning result.

17. The tangible, non-transient computer-readable medium of claim 16 having processor instructions to perform the method, wherein:

obtaining the additional information includes;

detecting a crossing event while implementing operations for lowering the unrecognized object toward the task location during implementation of the task, wherein the crossing event represents an entry and/or an exit of the unrecognized object relative to a laterally-oriented crossing reference located at a known height,

calculating an object height based on the crossing event, wherein the object height is used to derive the one or more scanning positions; and

implementing the operations for scanning the at least one portion of the unrecognized object while implementing the task includes;

at least temporarily stopping the implementation of the task,

implementing the operations for scanning the at least one portion of the unrecognized object while implementation of the task is temporarily stopped, and

lowering the unrecognized object toward the task location, after generating the scanning result, for completing the task.

18. The tangible, non-transient computer-readable medium of claim 16 having processor instructions to perform the method, wherein:

obtaining the additional information includes;

detecting a crossing event while implementing operations for lifting the unrecognized object from the start location during implementation of the task, wherein the crossing event represents an entry and/or an exit of the unrecognized object relative to a laterally-oriented crossing reference located at a known height;

calculating an object height based on the crossing event, wherein the object height is used to derive the one or more scanning positions; and

implementing the operations for scanning the at least one portion of the unrecognized object while implementing the task includes;

at least temporarily stopping the implementation of the task,

implementing the operations for scanning the at least one portion of the unrecognized object while implementation of the task is temporarily stopped, and

lowering the unrecognized object toward the task location, after generating the scanning result, for completing the task.

19. The tangible, non-transient computer-readable medium of claim 18 having processor instructions to perform the method, wherein:

the at least one portion of the unrecognized object includes a vertically-oriented surface;

implementing the operations for scanning the at least one portion of the unrecognized object includes implementing the operations, after detecting the crossing event, for presenting the vertically-oriented surface of the unrecognized object to a horizontally facing instance of the scanning sensor; and

generating the scanning result based on implementing the operations for scanning the vertically-oriented surface of the unrecognized object.

20. The tangible, non-transient computer-readable medium of claim 18 having processor instructions to perform the method, wherein:

the at least one portion of the unrecognized object includes a bottom surface;

implementing the operations for scanning the at least one portion of the unrecognized object includes implementing the operations, after detecting the crossing event, for presenting the bottom surface of the unrecognized object to an upward-facing instance of the scanning sensor; and

generating the scanning result based on implementing the operations for scanning the bottom surface of the unrecognized object.