Systems and methods for identifying and processing a variety of objects

• US 10,007,827 B2
• Filed: 09/09/2016
• Issued: 06/26/2018
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: Berkshire Grey Operating Co., Inc. (SoftBank Group Corp.)
• Original Assignee: Berkshire Grey, Inc. (SoftBank Group Corp.)
• Inventors: Wagner, Thomas, Ahearn, Kevin, Cohen, Benjamin, Dawson-Haggerty, Michael, Geyer, Christopher, Koletschka, Thomas, Maroney, Kyle, Mason, Matthew T., Price, Gene Temple, Romano, Joseph, Smith, Daniel, Srinivasa, Siddhartha, Velagapudi, Prasanna, Allen, Thomas
• Primary Examiner(s): Randazzo, Thomas
• Application Number:
  US15/260,837
• Publication Number:
  US 20170121113A1
• Time in Patent Office: 655 Days
• CPC Class Codes: B07C 5/36 Sorting apparatus characterised by the means used for distribution sorting according to destination B07C3/003, B07C3/02
• CPC Class Codes: B25J 19/022 using lasers
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1679 characterised by the tasks executed
• CPC Class Codes: B25J 9/1694 characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 1/1373 for fulfilling orders in warehouses
• CPC Class Codes: G05B 19/124 using tapes, cards or discs with optically sensed marks or codes G05B19/128, G05B19/14 take precedence
• CPC Class Codes: G05B 19/4183 characterised by data acquisition, e.g. workpiece identification
• CPC Class Codes: G05B 2219/14005 Alarm
• CPC Class Codes: G05B 2219/36371 Barcode reader
• CPC Class Codes: G05B 2219/45045 Maintenance, automatic storage and retrieval system
• CPC Class Codes: G05B 2219/45047 Sorting
• CPC Class Codes: G05B 2219/45063 Pick and place manipulator
• CPC Class Codes: G06K 19/06028 using bar codes
• CPC Class Codes: G06K 7/10693 for omnidirectional scanning
• CPC Class Codes: G06K 7/1404 Methods for optical code recognition
• CPC Class Codes: G06K 7/1443 locating of the code in an image
• CPC Class Codes: G06K 7/1447 extracting optical codes from image or text carrying said optical code
• CPC Class Codes: G06Q 10/087 : Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 2201/06 : Recognition of objects for industrial automation
• CPC Class Codes: G06V 40/1312 : direct reading, e.g. contactless acquisition
• CPC Class Codes: Y02P 90/02 : Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

A robotic system is disclosed that include an articulated arm and a first perception system for inspecting an object, as well as a plurality of additional perception systems, each of which is arranged to be directed toward a common area in which an object may be positioned by the robotic arm such that a plurality of views within the common area may be obtained by the plurality of additional perception systems.

Claims

1. An object processing system for identifying objects, said object processing system comprising:

a first identity capturing device for providing first identity data regarding a first view of an object at a first location;

a plurality of additional identity capturing devices, each of which is arranged to be directed toward a perception area for providing further identity data regarding a plurality of views of the object within the perception area, said first view and said plurality of views providing views of all surfaces of the object;

a computer processing system for receiving the first identity data and the further identity data, and for determining an identification of the object; and

an end effector for acquiring the object for movement from the first location to the perception area if the computer processing system cannot determine the identification of the object using only the first identity data.

2. The object processing system as claimed in claim 1, wherein the first identity capturing device includes a camera that is positioned above an area into which the objects are provided for identification and processing by the object processing system.

3. The object processing system as claimed in claim 1, wherein the plurality of additional identity capturing devices are positioned such that every surface of the object may be perceived regardless of the orientation of the object.

4. The object processing system as claimed in claim 1, wherein the first identity capturing device provides the first identity data regarding a unique object identifier, and wherein the plurality of additional identity capturing devices provide the further identity data that enables the computer processing system to perform any of confirming the unique object identifier or any additional object identifiers.

5. The object processing system as claimed in claim 1, wherein the first identity capturing device further provides object shape data regarding an object shape, and wherein the plurality of additional identity capturing devices provide the further identity data regarding a unique object identifier that is confirmed by the computer processing system as matching the object shape.

6. The object processing system as claimed in claim 1, wherein the object processing system further includes a routing system for receiving the object from the object identification system, and for routing the object in one of a plurality of directions responsive to at least one of the first identity data and the further identity data.

7. The object processing system as claimed in claim 1, wherein the perception area is defined by the plurality of additional perception units, and is generally in the shape of a bowl having a generally circular opening such that the end effector may introduce the object into the perception area through the generally circular opening.

8. The object processing system as claimed in claim 1, wherein the plurality of additional perception units are provided in a drop perception system having an open top through which an object may be dropped, and an open bottom through which the object may fall to a routing system for routing the object in one of a plurality of directions responsive to at least one of the first identity data and the further identity data.

9. The object processing system as claimed in claim 1, wherein the computer processing system determines a shape of the object.

10. The object processing system as claimed in claim 9, wherein the computer processing system generates a candidate pick list of objects to be selected from an inbound area.

11. The object processing system as claimed in claim 10, wherein the candidate pick list of objects ranks at least some of the objects corresponding to a desired pick order.

12. The object processing system as claimed in claim 1, wherein the first identity capturing device includes a scanner that is positioned above an area into which the objects are provided for identification and processing by the object processing system.

13. The object processing system as claimed in claim 12, wherein the scanner is a laser reflectivity scanner.

14. The object processing system as claimed in claim 12, wherein the scanner is a radio frequency ID scanner.

15. The object processing system as claimed in claim 1, wherein the plurality of additional identity capturing devices include a plurality of cameras.

16. The object processing system as claimed in claim 15, wherein the plurality of cameras includes five cameras that are collectively arranged to simulate holding the object in at least four different poses.

17. The object processing system as claimed in claim 15, wherein the plurality of cameras are provided in a drop camera unit into which an object may be dropped and scanned by the plurality of cameras as the object is falling through the unit.

18. The object processing system as claimed in claim 1, wherein the plurality of additional identity capturing devices include a plurality of scanners.

19. The object processing system as claimed in claim 18, wherein each of the plurality of scanners is a laser reflectivity scanner.

20. The object processing system as claimed in claim 18, wherein each of the plurality of scanners is a radio frequency ID scanner.

21. The object processing system as claimed in claim 18, wherein the plurality of scanners includes five scanners that are collectively arranged to simulate holding the object in at least four different poses.

22. The object processing system as claimed in claim 18, wherein the plurality of scanners are provided in a drop scanner unit into which an object may be dropped and scanned by the plurality of scanners as the object is falling through the unit.

23. A method of identifying an object in an object processing system including an end effector, said method comprising the steps of:

providing, by a first identity capturing device, first identity data regarding a first view of an object at a first location;

determining whether an identity of the object may be determined responsive to the first identity data, and if the identity of the object cannot be determined from the first identity data;

moving the object using the end effector to a plurality of additional identity capturing devices at a second location; and

providing further identity data regarding a plurality of views of the object within a perception area, by the plurality of additional identity capturing devices, each of which is arranged to be directed toward the object in the perception area from a different view of the plurality of views of the object within the perception area that may be obtained by the plurality of additional identity capturing devices, the first view of the object and the plurality of views of the object providing views of all surfaces of the object; and

processing any of the first identity data and the further identity data to identify the object.

24. The method as claimed in claim 23, wherein the method further includes the step of moving the object toward the first identity capturing device by a conveyor.

25. The method as claimed in claim 23, wherein the method further includes the step of moving the object toward the first identify capturing device together with a plurality of other, different objects.

26. The method as claimed in claim 23, wherein the step of moving the object to a plurality of additional identity capturing devices employs an articulated arm.

27. The method as claimed in claim 23, wherein the method further includes the step of verifying by using a computer processing system, whether an identification of a picked object that has been picked up by the end effector is the same as an identification of a selected object that was selected to be picked up by the end effector.

28. The method as claimed in claim 23, wherein the plurality of additional identity capturing devices are positioned such that together with the first view of the object, every surface of the object may be perceived regardless of the orientation of the object.

29. The method as claimed in claim 23, wherein the first identity capturing device provides the first identity data regarding a unique object identifier, and wherein the plurality of additional identity capturing devices provide the further identity data regarding any of confirming the object identity or any additional object identities.

30. The method as claimed in claim 23, wherein the first identity capturing device further provides object shape data regarding an object shape, and wherein the plurality of additional identity capturing devices provide additional the further identity data regarding a unique object identifier that is confirmed as matching the object shape.

31. The method as claimed in claim 23, wherein the plurality of additional perception units are arranged in a general shape of a bowl having a generally circular opening such that the end effector may introduce the object into the perception area through the generally circular opening.

32. The method as claimed in claim 23, wherein the plurality of additional perception units are provided in a drop perception system having an open top, and an open bottom, and wherein the method further includes the step of dropping the object into the drop perception unit, permitting the object to fall through the open bottom of the drop perception unit, receiving the object at a routing system, and directing the object in on of a plurality of directions using the routing system responsive to at least one of the first identity data and the further identity data.

33. The method as claimed in claim 23, wherein the method further includes the step of routing the object in one of a plurality of directions responsive to at least one of the first identity data and the further identity data.

34. The method as claimed in claim 23, wherein the method further includes the step of providing multi-object data regarding a plurality of additional objects near the object.

35. The method as claimed in claim 34, wherein the method further includes the step of providing supplemental multi-object data following the step of moving the object to confirm that the object was selected and moved.

36. The method as claimed in claim 23, wherein the plurality of additional identity capturing devices include a plurality of cameras.

37. The method as claimed in claim 36, wherein the plurality of cameras are provided in a drop camera unit, and the method further includes the steps of dropping an object into the drop camera unit and scanning by the plurality of cameras as the object is falling through the unit.

38. The method as claimed in claim 23, wherein the plurality of additional identity capturing devices include a plurality of scanners.

39. The method as claimed in claim 38, wherein the plurality of scanners are provided in a drop scanner unit, and the method further includes the steps of dropping an object into the drop scanner unit and scanning by the plurality of scanners as the object is falling through the unit.

40. An object identification system for identifying an object, said object identification system comprising a detection unit, a computer processing system and a routing system, said detection unit including a plurality of identity capturing devices that are each positioned to be directed toward a plurality of locations along an object path that the object may take as it travels into the object identification system, said plurality of identity capturing devices capturing identity data regarding all surfaces of the object, said computer processing system providing the identity of the object responsive to the identity data, and said routing system for receiving the object from the object identification system, and for routing the object in one of a plurality of directions responsive to the identity data.

41. The object identification system as claimed in claim 40, wherein the plurality of identity capturing devices include a plurality of cameras.

42. The object identification system as claimed in claim 40, wherein the plurality of identity capturing devices include a plurality of scanners.

43. The object identification system as claimed in claim 40, wherein the plurality of identity capturing devices include identity capturing devices at least at the bottom and central regions of the object identification system.

44. The object identification system as claimed in claim 40, wherein the plurality of identity capturing devices are arranged generally in the shape of a bowl having a generally circular opening such that the end effector may introduce the object into the perception area through the generally circular opening.

45. The object identification system as claimed in claim 40, wherein the plurality of identity capturing devices are provided in a drop perception system having an open top through which an object may be dropped, and an open bottom through which the object may fall.

46. The object identification system as claimed in claim 40, wherein the object identification system includes at least one sensor system for detecting whether an object has entered the detection unit.

47. The object identification system as claimed in claim 46, wherein the at least one sensor system includes a laser source and a laser sensor.

48. An object identification system comprising:

an end effector for acquiring an object for movement from a first location to a perception area;

a plurality of identity capturing devices, each of which is arranged to be directed toward the perception area in which the object may be positioned by the end effector such that identity data regarding a plurality of views within the perception area may be obtained by the plurality of identify capturing devices from distances from the perception area sufficient to detect identifying indicia on the object, said plurality of views including views of all surfaces of the object, said perception area being defined by the plurality of additional perception units, and being generally in the shape of a bowl having a generally circular opening such that the end effector may introduce the object into the perception area through the generally circular opening; and

a computer processing system for providing an identity of the object responsive to the identity data provided by the plurality of identity capturing devices.

49. The object identification system as claimed in claim 48, wherein the plurality of identity capturing devices includes a plurality of cameras.

50. The object identification system as claimed in claim 48, wherein the plurality of identity capturing devices includes a plurality of scanners.

51. The object identification system as claimed in claim 48, wherein the system further includes a routing system for receiving the object from the object identification system, and for routing the object in one of a plurality of directions responsive to the identity data.

52. The object identification system as claimed in claim 48, wherein the plurality of identity capturing devices encircle the perception area in three dimensions.

53. The object identification system as claimed in claim 52, wherein the plurality of identity capturing devices are positioned a fixed locations.

54. A method of identifying an object in an object processing system including an end effector, said method comprising the steps of:

providing, by a first identity capturing device, first identity data regarding a first view of an object at a first location;

moving the object using the end effector from the first location to a plurality of additional identity capturing devices at a second location;

providing further identity data regarding a plurality of views of the object within a perception area at the second location, by the plurality of additional identity capturing devices, each of which is arranged to be directed toward the object in the perception area from a different view of the plurality of views of the object within the perception area that may be obtained by the plurality of additional identity capturing devices, the first view of the object and the plurality of views of the object providing views of all surfaces of the object;

processing the first identity data and the further identity data to identify the object; and

routing the object responsive to the determined identity of the object in one of a plurality of directions.

55. The method as claimed in claim 54, wherein the method further includes the step of moving the object toward the first identity capturing device by a conveyor.

56. The method as claimed in claim 54, wherein the step of moving the object to a plurality of additional identity capturing devices employs an articulated arm.

57. The method as claimed in claim 54, wherein the method further includes the step of verifying by using a computer processing system, whether an identification of a picked object that has been picked up by the end effector is the same as an identification of a selected object that was selected to be picked up by the end effector.

58. The method as claimed in claim 54, wherein the plurality of identity capturing devices are arranged generally in the shape of a bowl having a generally circular opening such that the end effector may introduce the object into the perception area through the generally circular opening.

59. The object identification system as claimed in claim 54, wherein the plurality of identity capturing devices are provided in a drop perception system having an open top, and an open bottom, and wherein the method further includes the step of dropping the object into the drop perception unit and permitting the object to fall through the open bottom of the drop perception unit.

60. The method as claimed in claim 54, wherein the plurality of additional identity capturing devices include a plurality of cameras.

61. The method as claimed in claim 60, wherein the plurality of cameras are provided in a drop camera unit, and the method further includes the steps of dropping an object into the drop camera unit and scanning by the plurality of cameras as the object is falling through the unit.

62. The method as claimed in claim 54, wherein the plurality of additional identity capturing devices include a plurality of scanners.

63. The method as claimed in claim 62, wherein the plurality of scanners are provided in a drop scanner unit, and the method further includes the steps of dropping an object into the drop scanner unit and scanning by the plurality of scanners as the object is falling through the unit.

64. A method of identifying an object in an object processing system, said method comprising the steps of:

providing, by a first identity capturing device, first identity data regarding a first view of an object at a first location;

dropping the object into a drop perception unit at a second location;

providing further identity data regarding a plurality of views of the object within the drop perception unit by a plurality of additional identity capturing devices, each of which is arranged to be directed toward the object within the drop perception unit from a different view of the plurality of views of the object within the drop perception unit that may be obtained by the plurality of additional identity capturing devices, the first view of the object and the plurality of views of the object providing views of all surfaces of the object;

processing the first identity data and the further identity data to identify the object; and

routing the object responsive to the determined identity of the object in one of a plurality of directions.

65. The method as claimed in claim 64, wherein the method further includes the step of moving the object toward the first identity capturing device by a conveyor.

66. The method as claimed in claim 64, wherein the method further includes the step of moving the object toward the first identify capturing device together with a plurality of other, different objects.

67. The method as claimed in claim 64, wherein the step of moving the object to a plurality of additional identity capturing devices employs an articulated arm.

68. The method as claimed in claim 64, wherein the plurality of additional identity capturing devices include a plurality of cameras.

69. The method as claimed in claim 64, wherein the plurality of additional identity capturing devices include a plurality of scanners.

70. The method as claimed in claim 64, wherein the plurality of additional identity capturing devices are positioned such that every surface of the object may be perceived regardless of the orientation of the object.