3D-2D vision system for robotic carton unloading

• US 10,315,866 B2
• Filed: 04/07/2017
• Issued: 06/11/2019
• Est. Priority Date: 10/20/2016
• Status: Active Grant

• Current Assignee: Intelligrated Incorporated (Honeywell International Inc.)
• Original Assignee: Intelligrated Incorporated (Honeywell International Inc.)
• Inventors: Yuvaraj, Karthikeyan, Tankasala, Sriram Pavan, Nilson, Benjamin E., Zhang, Feng, Wicks, Matthew R.
• Primary Examiner(s): MacKey, Patrick H
• Application Number:
  US15/481,969
• Publication Number:
  US 20180111769A1
• Time in Patent Office: 795 Days
• CPC Class Codes: B25J 15/00 Gripping heads and other end effectors grippers used in machine tools B23Q7/04; gripping members fitted on cranes B66C1/42, B66C1/44; gripping means used in the manufacture of semiconductors H01L21/68707; gripping means used for mounting electrical components H05K13/04
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 67/02 Loading or unloading land vehicles
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: G05B 2219/40442 Voxel map, 3-D grid map
• CPC Class Codes: G05B 2219/45056 Handling cases, boxes

Abstract

Robotic carton loader or unloader incorporates three-dimensional (3D) and two-dimensional (2D) sensors to detect respectively a 3D point cloud and a 2D image of a carton pile within transportation carrier such as a truck trailer or shipping container. Edge detection is performed using the 3D point cloud, discarding segments that are two small to be part of a product such as a carton. Segments that are too large to correspond to a carton are 2D image processed to detect additional edges. Results from 3D and 2D edge detection are converted in a calibrated 3D space of the material carton loader or unloader to perform one of loading or unloading of the transportation carrier. Image processing can also detect jamming of products sequence from individually controllable zones of a conveyor of the robotic carton loader or unloader for singulated unloading.

Claims

1. A method of determining locations of individual cartons in a material handling system, the method comprising:

receiving, by a carton detection system, a two-dimensional (2D) image and a three-dimensional (3D) point cloud of at least one portion of a carton pile resting on a floor of a transportation carrier;

detecting, by a processing subsystem in connection with the carton detection system, segments within the 3D point cloud;

removing, by the processing subsystem, any segments that are smaller than a first threshold;

determining, by the processing subsystem, whether any segments are less than a second threshold;

in response to determining that a selected segment is less than the second threshold, qualifying the selected segment as a 3D detected carton;

in response to determining that a selected segment is not less than the second threshold;

determining, by the processing subsystem, a 2D mask that corresponds to the selected segment;

determining, by the processing subsystem, a portion of the 2D image that corresponds to the 2D mask;

detecting, by the processing subsystem, segments within the portion of the 2D image; and

qualifying, by the processing subsystem, detected segments as 2D detected cartons;

combining, by the processing subsystem, the 2D and 3D detected cartons in a detection result; and

converting, by the processing subsystem, the detection result using calibration information into 3D locations relative to a robotic carton handling system for a selected one of a loading operation and an unloading operation performed by a robotic carton manipulator of the robotic carton handling system.

2. The method of claim 1, further comprising:

receiving the 2D image and 3D point cloud from one or more sensors positioned on the robotic carton handling system to detect the one portion of the carton pile;

receiving a 2D image and 3D point cloud from another one or more sensors positioned on the robotic carton handling system to detect a contiguous portion of the carton pile; and

combining the 2D and 3D detected cartons from both the one portion and the contiguous portion to form the detection result.

3. The method of claim 1, wherein qualifying detected segments as 2D detected cartons further comprises discarding, by the processing subsystem, edges that form a smaller rectangle fully encompassed within a larger rectangle.

4. The method of claim 1, further comprising:

performing the selected one of the loading operation and the unloading operation by the robotic carton handling system using the 3D locations.

5. A robotic carton handling system for unloading cartons in a carton pile, the robotic carton handling system movable across a floor, the robotic carton handling system comprising:

a mobile body;

a movable robotic manipulator attached to the mobile body and comprising an end effector at an end thereof, the end effector configured to unload one or more cartons from the carton pile;

a conveyor mounted on the mobile body configured to receive the one or more cartons from the end effector and to move the one or more cartons towards a rear of the robotic carton handling system;

a carton detection system comprising;

one or more sensors coupled respectively to one of the mobile body and the movable robotic manipulator to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of at least one portion of a carton pile resting on a floor of a transportation carrier;

a processing subsystem in communication with the one or more sensors, the processing subsystem;

detects segments within the 3D point cloud;

removes any segments that are smaller than a first threshold;

determines whether any segments are less than a second threshold;

in response to determining that a selected segment is less than the second threshold, qualifies the selected segment as a 3D detected carton;

in response to determining that a selected segment is not less than the second threshold;

determines a 2D mask that corresponds to the selected segment;

determines a portion of the 2D optical image that corresponds to the 2D mask;

detects segments within the portion of the 2D optical image; and

qualifies detected segments as 2D detected cartons;

combines the 2D and 3D detected cartons in a detection result; and

converts the detection result using calibration information into 3D locations relative to a robotic carton handling system for a selected one of a loading operation and an unloading operation.

6. The robotic carton handling system of claim 5, wherein:

the one or more sensors are positioned on the robotic carton handling system to detect the one portion of the carton pile;

the robotic carton handling system further comprises another one or more sensors positioned on the robotic carton handling system to detect a contiguous portion of the carton pile; and

the processor subsystem combines the 2D and 3D detected cartons from both the one portion and the contiguous portion to form the detection result.

7. The robotic carton handling system of claim 5, wherein the processor subsystem qualifies detected segments as 2D detected cartons at least in part by discarding edges that form a smaller rectangle fully encompassed within a larger rectangle.

8. The robotic carton handling system of claim 5, further comprising an automation controller in communication with the processing subsystem, the automation controller causes the robotic carton manipulator to perform the selected one of the loading operation and the unloading operation by the robotic carton handling system using the 3D locations.

9. A material handling system comprising:

a robotic carton handling system for unloading cartons in a carton pile, the robotic carton handling system movable across a floor, the robotic carton handling system comprising;

a mobile body;

a movable robotic manipulator attached to the mobile body and comprising an end effector at an end thereof, the end effector configured to unload one or more cartons from the carton pile;

a conveyor mounted on the mobile body configured to receive the one or more cartons from the end effector and to move the one or more cartons towards a rear of the robotic carton handling system;

a carton detection system comprising;

one or more sensors coupled respectively to one of the mobile body and the movable robotic manipulator to provide a two-dimensional (2D) optical image and a three-dimensional (3D) point cloud of at least one portion of a carton pile resting on a floor of a transportation carrier;

a processing subsystem in communication with the one or more sensors, the processing subsystem;

detects segments within the 3D point cloud;

removes any segments that are smaller than a first threshold;

determines whether any segments are less than a second threshold;

in response to determining that a selected segment is less than the second threshold, qualifies the selected segment as a 3D detected carton;

in response to determining that a selected segment is not less than the second threshold;

determines a 2D mask that corresponds to the selected segment;

determines a portion of the 2D optical image that corresponds to the 2D mask;

detects segments within the portion of the 2D optical image; and

qualifies detected segments as 2D detected cartons;

combines the 2D and 3D detected cartons in a detection result; and

converts the detection result using calibration information into 3D locations relative to a robotic carton handling system for a selected one of a loading operation and an unloading operation;

an automation controller in communication with the processing subsystem, the automation controller causes the robotic carton manipulator to perform the selected one of the loading operation and the unloading operation by the robotic carton handling system using the 3D locations; and

an extendable conveyor system having a proximal end coupled to a stationary conveyor system and a movable distal end positioned proximate to the robotic carton handling system to transfer cartons between the stationary conveyor and the robotic carton handling system.