Truck Unloader Visualization

• US 20150352721A1
• Filed: 06/04/2015
• Published: 12/10/2015
• Est. Priority Date: 06/04/2014
• Status: Active Grant

• Current Assignee: Intelligrated Incorporated (Honeywell International Inc.)
• Original Assignee: Intelligrated Incorporated (Honeywell International Inc.)
• Inventors: McCollum, Joshua, Wicks, Matthew R., Nilson, Benjamin, Yuvarej, Karthikeyan, Mukherjee, Prasenjit, Packard, Daniel
• Granted Patent: US 9,744,669 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1679 characterised by the tasks executed
• CPC Class Codes: B25J 9/1687 Assembly, peg and hole, palletising, straight line, weaving pattern movement
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 67/24 Unloading land vehicles
• CPC Class Codes: G05B 2219/40006 Placing, palletize, un palletize, paper roll placing, box stacking
• CPC Class Codes: G05B 2219/45056 Handling cases, boxes
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: Y10S 901/02 Arm motion controller
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods, devices, systems, and non-transitory process-readable storage media for a computing device of a robotic carton unloader to identify items to be unloaded from an unloading area within imagery.

Claims

1. A method for controlling a robotic carton unloader, comprising:

obtaining image data from a first sensor of the robotic carton unloader of a carton pile;

detecting cartons in a face of the carton pile in the image data from the first sensor;

sending indications of the detected cartons; and

controlling a robotic arm of the robotic carton unloader to unload the detected cartons onto a conveyor of the robotic carton unloader based on the indications of the detected cartons.

2. The method of claim 1, wherein:

obtaining image data from a first sensor of the robotic carton unloader of a carton pile comprises obtaining image data from a first sensor of the robotic carton unloader of a carton pile in a first computing device of the robotic carton unloader;

detecting cartons in a face of the carton pile in the image data from the first sensor comprises detecting cartons in a face of the carton pile in the image data from the first sensor in the first computing device;

sending indications of the detected cartons comprises sending indications of the detected cartons from the first computing device to a second computing device of the robotic carton unloader; and

controlling a robotic arm of the robotic carton unloader to unload the detected cartons onto a conveyor of the robotic carton unloader based on the indications of the detected cartons comprises controlling, by the second computing device, a robotic arm of the robotic carton unloader to unload the detected cartons onto a conveyor of the robotic carton unloader based on the indications of the detected cartons

3. The method of claim 2, wherein detecting cartons in the face of the carton pile in the image data from the first sensor in the first computing device comprises performing, by the first computing device, edge detection operations to identify edges of cartons in the face of the carton pile.

4. The method of claim 2, wherein detecting cartons in the face of the carton pile in the image data from the first sensor in the first computing device comprises detecting one or more of a hole, a label, and a logo in the image data for each detected carton.

5. The method of claim 2, further comprising:

obtaining image data from a second sensor of the robotic carton unloader of the carton pile in a third computing device of the robotic carton unloader;

detecting cartons in the face of the carton pile in the image data from the second sensor in the third computing device;

obtaining image data from a third sensor of the robotic carton unloader of the carton pile in a fourth computing device of the robotic carton unloader;

detecting cartons in the face of the carton pile in the image data from the third sensor in the fourth computing device;

sending indications of the detected cartons from the third computing device and the fourth computing device to the first computing device; and

combining the indications of the detected cartons from the first computing device, the third computing device, and the fourth computing device together to generate a master set of indications of detected cartons at the first computing device,

wherein sending indications of the detected cartons from the first computing device to the second computing device of the robotic carton unloader comprises sending the master set of indications of detected cartons from the first computing device to the second computing device of the robotic carton unloader.

6. The method of claim 5, wherein the image data from the first sensor, the image data from the second sensor, and the image data from the third sensor are RGB data and depth data.

7. The method of claim 6, wherein the first sensor, the second sensor, and the third sensors are Microsoft Kinect sensors.

8. The method of claim 1, wherein controlling the robotic arm of the robotic carton unloader to unload the detected cartons onto the conveyor of the robotic carton unloader based on the indications of the detected cartons comprises controlling the robotic arm of the robotic carton unloader to unload the detected cartons onto the conveyor of the robotic carton unloader based on the indications of the detected cartons without triggering the first sensor to obtain additional image data until all the detected cartons are unloaded.

9. The method of claim 8, further comprising:

determining that all the detected cartons are unloaded; and

triggering the first sensor to obtain the additional image data of the carton pile to detect cartons in a next face of the carton pile in response to determining that all detected cartons are unloaded.

10. A robotic carton unloader, comprising:

a vision system, comprising;

a first sensor; and

a first processor connected to the first sensor;

a conveyor;

a robotic arm; and

a second processor connected to the robotic arm and the first processor of the vision system,

wherein the first processor of the vision system is configured with processor executable instructions to perform operations comprising;

obtaining image data from the first sensor of a carton pile;

detecting cartons in a face of the carton pile in the image data from the first sensor; and

sending indications of the detected cartons to the second processor, and

wherein the second processor is configured with processor executable instructions to perform operations comprising controlling the robotic arm to unload the detected cartons onto the conveyor based on the received indications of the detected cartons.

11. The robotic carton unloader of claim 10, wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that detecting cartons in the face of the carton pile in the image data from the first sensor comprises performing edge detection operations to identify edges of cartons in the face of the carton pile.

12. The robotic carton unloader of claim 10, wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that detecting cartons in the face of the carton pile in the image data from the first sensor comprises detecting one or more of a hole, a label, and a logo in the image data for each detected carton.

13. The robotic carton unloader of claim 10, wherein the vision system further comprises:

a second sensor;

a third processor connected to the second sensor and the first processor, wherein the third processor of the vision system is configured with processor executable instructions to perform operations comprising;

obtaining image data from the second sensor of the carton pile;

detecting cartons in the face of the carton pile in the image data from the second sensor; and

sending indications of the detected cartons to the first processor;

a third sensor; and

a fourth processor connected to the third sensor and the first processor, wherein the fourth processor of the vision system is configured with processor executable instructions to perform operations comprising;

obtaining image data from the third sensor of the carton pile;

detecting cartons in the face of the carton pile in the image data from the third sensor; and

sending indications of the detected cartons to the first processor,

wherein the first processor of the vision system is configured with processor executable instructions to perform operations further comprising combining the indications of the detected cartons from the first processor, the third processor, and the fourth processor together to generate a master set of indications of detected cartons at the first processor, and

wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that sending indications of the detected cartons to the second processor comprises sending the master set of indications of detected cartons to the second processor.

14. The robotic carton unloader of claim 13, wherein the image data from the first sensor, the image data from the second sensor, and the image data from the third sensor are RGB data and depth data.

15. The robotic carton unloader of claim 14, wherein the first sensor, the second sensor, and the third sensors are Microsoft Kinect sensors.

16. The robotic carton unloader of claim 10, wherein the second processor is configured with processor executable instructions to perform operations such that controlling the robotic arm to unload the detected cartons onto the conveyor based on the received indications of the detected cartons comprises controlling the robotic arm of the robotic carton unloader to unload the detected cartons based on the indications of the detected cartons without triggering the first sensor to obtain additional image data until all the detected cartons are unloaded.

17. The robotic carton unloader of claim 16, wherein the second processor is configured with processor executable instructions to perform operations further comprising:

determining that all the detected cartons are unloaded; and

triggering the first sensor to obtain the additional image data of the carton pile to detect cartons in a next face of the carton pile in response to determining that all detected cartons are unloaded.

18. The robotic carton unloader of claim 17, wherein:

the first processor of the vision system is configured with processor executable instructions to perform operations comprising;

obtaining the additional image data from the first sensor of the carton pile in response to the triggering of the first sensor to obtain the additional image data of the carton pile to detect cartons in the next face of the carton pile;

detecting additional cartons in the next face of the carton pile in the additional image data from the first sensor; and

sending indications of the detected additional cartons to the second processor, and

wherein the second processor is configured with processor executable instructions to perform operations comprising controlling the robotic arm to unload the detected additional cartons onto the conveyor based on the received indications of the detected additional cartons

19. A vision system for a robotic carton unloader, comprising:

a first sensor; and

a first processor connected to the first sensor, wherein the first processor of the vision system is configured with processor executable instructions to perform operations comprising;

receiving a trigger to obtain image data from a computing device of the robotic carton unloader;

obtaining image data from the first sensor of a carton pile;

detecting cartons in a face of the carton pile in the image data from the first sensor; and

sending indications of the detected cartons to the computing device of the robotic carton unloader to enable the computing device to control a robotic arm of the carton unloader to unload the detected cartons based on the received indications of the detected cartons.

20. The vision system of claim 19, wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that detecting cartons in the face of the carton pile in the image data from the first sensor comprises performing edge detection operations to identify edges of cartons in the face of the carton pile.

21. The vision system of claim 19, wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that detecting cartons in the face of the carton pile in the image data from the first sensor comprises detecting one or more of a hole, a label, and a logo in the image data for each detected carton.

22. The vision system of claim 19, further comprising:

a second sensor;

a second processor connected to the second sensor and the first processor, wherein the second processor of the vision system is configured with processor executable instructions to perform operations comprising;

obtaining image data from the second sensor of the carton pile;

detecting cartons in the face of the carton pile in the image data from the second sensor; and

sending indications of the detected cartons to the first processor;

a third sensor; and

a third processor connected to the third sensor and the first processor, wherein the third processor of the vision system is configured with processor executable instructions to perform operations comprising;

obtaining image data from the third sensor of the carton pile;

detecting cartons in the face of the carton pile in the image data from the third sensor; and

sending indications of the detected cartons to the first processor,

wherein the first processor of the vision system is configured with processor executable instructions to perform operations further comprising combining the indications of the detected cartons from the first processor, the second processor, and the third processor together to generate a master set of indications of detected cartons at the first processor, and

wherein the first processor of the vision system is configured with processor executable instructions to perform operations such that sending indications of the detected cartons to the computing device of the robotic carton unloader comprises sending the master set of indications of detected cartons to the computing device of the robotic carton unloader.

23. The vision system of claim 22, wherein the image data from the first sensor, the image data from the second sensor, and the image data from the third sensor are RGB data and depth data.

24. The vision system of claim 23, wherein the first sensor, the second sensor, and the third sensors are Microsoft Kinect sensors.