Automated package registration systems, devices, and methods

• US 10,562,188 B1
• Filed: 06/17/2019
• Issued: 02/18/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: MUJIN Inc.
• Original Assignee: MUJIN Inc.
• Inventors: Diankov, Rosen, Liu, Huan, Ye, Xutao, Rodrigues, Jose Jeronimo Moreira, Kanemoto, Yoshiki, Yu, Jinze, Islam, Russell
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US16/443,743
• Time in Patent Office: 246 Days
• CPC Class Codes: B25J 13/085 Force or torque sensors B25J13/082, B25J13/084 take precedence
• CPC Class Codes: B25J 15/0052 multiple gripper units or multiple end effectors
• CPC Class Codes: B25J 15/0616 with vacuum
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 47/91 incorporating pneumatic, e.g. suction, grippers
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: G05B 2219/39001 Robot, manipulator control
• CPC Class Codes: G05B 2219/40067 Stack irregular packages
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A method of operating a package handling system includes: receiving first image data representative of a first surface image of a package surface; identifying a pair of intersecting edges for the package surface based on the first image; determining a minimum viable region based on the pair of edges, the minimum viable region for gripping and lifting the package; receiving second image data representative of the package after the lift; and creating registration data based on the third image data.

Claims

1. A method for registering a package during a robotic package handling operation, the method comprising:

receiving first image data representative of a first surface image of a package surface;

determining that the package is unregistered based on comparing the first surface image to a registration record;

identifying a pair of edges for the package surface based on the first image data, wherein the pair of edges are exposed along horizontal directions and intersect each other to form a corner of the package surface;

estimating unclear edges based on the first image data, wherein the pair of edges and the unclear edges correspond to boundaries of the package surface;

identifying a straight line based on the first image data;

determining a minimum viable region based on the pair of edges, wherein the minimum viable region overlaps and/or coincides with one or both of the pair of edges and wherein the minimum viable region is a continuous region that excludes the straight line, wherein the minimum viable region does not overlap with the unclear edges for gripping the package while leaving exposed portions of the package corresponding to the estimated unclear edges;

generating one or more commands for (1) gripping the package with an end effector positioned over the minimum viable region and (2) lifting the package;

receiving second image data representative of the package after the lift;

determining third image data as a function of the first image data and the second image data; and

creating registration data representative of a new registration record of the package based on the third image data for subsequently identifying other packages.

2. The method of claim 1, further comprising:

identifying one or more new edges based on the second image data, wherein the new edges represent actual exposed edges of the package corresponding to the estimated unclear edges; and

wherein;

the third image data includes the one or more new edges.

3. The method of claim 2, wherein identifying the one or more new edges includes identifying the one or more new edges based on differences in depth data measured after lifting the package.

4. The method of claim 1, wherein determining the third image data includes:

calculating a difference between the first image data and the second image data; and

combining the first image data with the calculated difference.

5. The method of claim 1, further comprising:

calculating a height representing a dimension of the package along a vertical direction; and

wherein;

the registration data includes the calculated height.

6. The method of claim 5, wherein calculating the height includes:

detecting a trigger representing a timing when the package crosses a horizontal detection plane associated with a height determination sensor located at a known height;

determining a vertical position associated with the trigger, wherein the vertical position represents a vertical location of the end effector at the timing when the package crosses the horizontal detection plane; and

calculating the height based on a difference between the known height and the vertical position.

7. The method of claim 1, further comprising:

receiving data after lifting the package, wherein the data represents a force measurement from an F-T sensor connected to or integral with the end effector;

determining a weight of the package based on the received data; and

wherein;

the registration data includes the determined weight.

8. The method of claim 1, further comprising:

receiving data after lifting the package, wherein the data represents a torque measurement from an F-T sensor connected to or integral with the end effector;

determining a center of mass location of the package based on the received data; and

wherein;

the registration data includes the center of mass location.

19. The method of claim 6, wherein the trigger represents the time when the package crosses the horizontal detection plane directly above and overlapping a destination location.

9. A system for registering an unregistered package during a robotic package handling operation, the system comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the memory including instructions executable by the at least one processor to;

receive first image data representative of a first surface image of a package surface;

determine that the package is unregistered based on comparing the first surface image to a registration record;

identify a pair of edges for the package surface based on the first image data, wherein the pair of edges are exposed along horizontal directions and intersect each other to form a corner of the package surface;

estimate unclear edges based on the first image data, wherein the pair of edges and the unclear edges correspond to boundaries of the package surface;

identify a straight line based on the first image data;

determine a minimum viable region based on the pair of edges, wherein the minimum viable region overlaps and/or coincides with one or both of the pair of edges and wherein the minimum viable region is a continuous region that excludes the straight line, wherein the minimum viable region does not overlap with the unclear edges for gripping the package while leaving exposed portions of the package corresponding to the estimated unclear edges;

generate one or more commands for (1) gripping the package with an end effector positioned over the minimum viable region and (2) lifting the package;

receive second image data of the package, wherein the second image data represents the package after the lift;

determine third image data as a function of the first image data and the second image data; and

create registration data representative of a new registration record of the package based on the third image data for subsequently identifying other packages.

10. The system of claim 9, wherein the at least one memory includes instructions to:

receive data after lifting the package, wherein the data represents a force measurement and/or a torque measurement from an F-T sensor connected to or integral with the end effector; and

wherein;

the registration data includes the received data or a processing result thereof representative of a weight and/or a center of mass location of the package.

11. The system of claim 9, wherein the at least one memory includes instructions to:

calculate a height representing a dimension of the package along a vertical direction; and

wherein;

the registration data includes the calculated height.

12. The system of claim 11, wherein calculating the height includes:

detecting a trigger representing a timing when the package crosses a horizontal detection plane associated with a height determination sensor located at a known height;

determining a vertical position associated with the trigger, wherein the vertical position represents a vertical location of the end effector at the timing when the package crosses the horizontal detection plane; and

calculating the height based on a difference between the known height and the vertical position.

13. The system of claim 9, wherein the at least one memory includes instructions to determine the third image data based on:

calculating a difference between the first image data and the second image data; and

combining the first image data with the calculated difference.

14. A non-transitory memory medium storing computer-executable instructions, when executed by a computing system, cause the computing system to perform a method, the method comprising:

receiving first image data representative of a first surface image of a package surface;

determining that the package is unregistered based on comparing the first surface image to a registration record;

identifying a pair of edges for the package surface based on the first image data, wherein the pair of edges are exposed along horizontal directions and intersect each other to form a corner of the package surface;

estimating unclear edges based on the first image data, wherein the pair of edges and the unclear edges correspond to boundaries of the package surface;

identifying a straight line based on the first image data;

determining a minimum viable region based on the pair of edges, wherein the minimum viable region overlaps and/or coincides with one or both of the pair of edges and wherein the minimum viable region is a continuous region that excludes the straight line, wherein the minimum viable region does not overlap with the unclear edges for gripping the package while leaving exposed portions of the package corresponding to the estimated unclear edges;

generating one or more commands for (1) gripping the package with an end effector positioned over the minimum viable region and (2) lifting the package;

receiving second image data of the package, wherein the second image data represents the package after the lift;

determining third image data as a function of the first image data and the second image data; and

creating registration data representative of a new registration record of the package based on the third image data for subsequently identifying other packages.

15. The non-transitory memory medium of claim 14, wherein the method further comprises:

receiving data after lifting the package, wherein the data represents a force measurement and/or a torque measurement from an F-T sensor connected to or integral with the end effector; and

wherein;

the registration data includes the received data or a processing result thereof representative of a weight and/or a center of mass location of the package.

16. The non-transitory memory medium of claim 14, wherein the method further comprises:

calculating a height representing a dimension of the package along a vertical direction; and

wherein;

the registration data includes the calculated height.

17. The non-transitory memory medium of claim 16, wherein the calculating the height includes:

detecting a trigger representing a timing when the package crosses a horizontal detection plane associated with a height determination sensor located at a known height;

determining a vertical position associated with the trigger, wherein the vertical position represents a vertical location of the end effector at the timing when the package crosses the horizontal detection plane; and

calculating the height based on a difference between the known height and the vertical position.

18. The non-transitory memory medium of claim 14, wherein the determining the third image data includes:

calculating a difference between the first image data and the second image data; and

combining the first image data with the calculated difference.