SYSTEMS AND METHODS FOR LEARNING TO EXTRAPOLATE OPTIMAL OBJECT ROUTING AND HANDLING PARAMETERS

• US 20200130935A1
• Filed: 10/25/2019
• Published: 04/30/2020
• Est. Priority Date: 10/25/2018
• Status: Active Grant

• Current Assignee: Berkshire Grey Operating Co., Inc. (SoftBank Group Corp.)
• Original Assignee: Berkshire Grey, Inc. (SoftBank Group Corp.)
• Inventors: Wagner, Thomas, Mason, Matthew T., Koletschka, Thomas, Schneider, Abraham, Javdani, Shervin, Geyer, Christopher
• Granted Patent: US 11,407,589 B2
• Time in Patent Office: Days
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 1/1371 with data records
• CPC Class Codes: B65G 1/1373 for fulfilling orders in warehouses
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: G05B 19/41 characterised by interpolation, e.g. the computation of intermediate points between programmed end points to define the path to be followed and the rate of travel along that path G05B19/25, G05B19/31, G05B19/37, G05B19/39, G05B19/40 take precedence
• CPC Class Codes: G05B 2219/39298 Trajectory learning
• CPC Class Codes: G05B 2219/40053 Pick 3-D object from pile of objects
• CPC Class Codes: G05B 2219/45063 Pick and place manipulator
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

A system for object processing is disclosed. The system includes a framework of processes that enable reliable deployment of artificial intelligence-based policies in a warehouse setting to improve the speed, reliability, and accuracy of the system. The system harnesses a vast number of picks to provide data points to machine learning techniques. These machine learning techniques use the data to refine or reinforce in-use policies to optimize the speed and successful transfer of objects within the system. For example, objects in the system are identified at a supply location, a predetermined set of information regarding object is retrieved and combined with a set of object information and processing parameters determined by the system. The combined information is then used to determine routing of the object according to an initial policy. This policy is then observed, altered, tested, and re-implemented in an altered form.

Claims

1. A system for object processing, said system comprising:

a supply location having one or more objects to be processed;

a plurality of object processing stations for processing objects;

an object classification system, wherein the object classification system includes;

an identification system for identifying an object at the supply location;

a first data repository having a first set of object information; and

a second data repository having a second set of object information, said second set of object information including processing parameters;

wherein said first set of object information is supplied with the object and the second set of object information is determined by the system; and

wherein the object classification system assigns the object to a class based on the first set of information and the second set of information; and

a routing system for routing an object from the supply location to a selected object processing station capable of processing objects in the assigned class, wherein the selected object processing station processes the object according to the determined processing parameters.

2. The system as claimed in claim 1 further comprising an evaluation system for evaluating the interaction of the object processing station with the object according to the processing parameters, wherein the evaluation system determines whether the interaction was successful or unsuccessful.

3. The system as claimed in claim 2 wherein the evaluation system determines the interaction success by evaluating object grasping, object movement, and object placement.

4. The system as claimed in claim 3 wherein the evaluation relies on sensor input.

5. The system as claimed in claim 3 wherein the evaluation relies on human input.

6. The system as claimed in claim 2 wherein the processing parameters are updated in response to the determination of the evaluation system.

7. The system as claimed in claim 1, wherein the second set of object information includes feature information regarding object features.

8. The system as claimed in claim 7, wherein the feature information is developed associated with any of object stockkeeping unit (SKU), text description, product category, object manufacturer, object mass, object material, object shape, packaging details, object images, and object color.

9. The system as claimed in claim 7, wherein the feature information is developed associated with any of sensed information, object weight, object material, object shape, object opacity, object size, object volume, effector to be used, grip location, object movement limitations, object fragility, and object processing limitations.

10. A method of processing objects comprising the steps of:

presenting an object to a routing station;

evaluating provided object information and determined object information associated with the object, wherein the determined object information includes processing parameters;

classifying the object based on provided and determined object information;

selecting a picking station to process the object based on object classification;

routing the object from the routing station to the selected picking station;

processing the object at the picking station according to said processing parameters;

observing the object being processed at the processing station;

scoring the object handling performance based on observed criteria; and

updating the processing parameters based on object handling performance score.

11. The method as claimed in claim 10 wherein the step of determining includes receiving input from one or more sensors.

12. The method as claimed in claim 11, wherein the input from the one or more sensors is associated with the processing parameters.

13. The method as claimed in claim 10 wherein the step of determining includes human input.

14. The method as claimed in claim 10, wherein the classifying step includes comparing provided and determined object information of the presented object with provided and determined object information of previously presented objects.

15. The method as claimed in claim 10, wherein the step of observing uses sensory information to determine a successful interaction or a failed interaction.

16. The method as claimed in claim 15 wherein the sensory information comes from one or more of depth sensors, scanners, cameras (e.g., RGB), flow sensors, pressure sensors, position sensors, and force sensors, mass (and/or weight) scales, acceleration sensors, and vibration sensors.

17. The method as claimed in claim 15 wherein the sensory information comes from a human observer.

18. The method as claimed in claim 10, wherein the step of scoring includes evaluating initial grasp failure, object transit failure, object placement failure, and object damage.

19. The method as claimed in claim 10 wherein the selected picking station can include a robotic picker.

20. The method as claimed in claim 10 wherein the selected picking station can include a human picker.

21. The method as claimed in claim 10, wherein the determined object information includes feature information regarding object features.

22. The method as claimed in claim 21, wherein the feature information is developed associated with any of object stockkeeping unit (SKU), text description, product category, object manufacturer, object mass, object material, object shape, packaging details, object images, and object color.

23. The method as claimed in claim 21, wherein the feature information is developed associated with any of sensed information, object weight, object material, object shape, object opacity, object size, object volume, effector to be used, grip location, object movement limitations, object fragility, and object processing limitations.

24. A method of processing objects, said method comprising the steps of:

providing, at a supply location, one or more objects to be processed;

providing an object classification system, wherein the object classification system includes;

an identification system for identifying an object at the supply location;

a first data repository having a first set of object information; and

a second data repository having a second set of object information, said second set of object information including feature information regarding a plurality of objects;

providing the first set of object information with the object;

providing the second set of object information via a feedback learning system, wherein the second set of object information includes feature information regarding object features;

assigning the object to a class based on the first set of information and the second set of information; and

updating the second set of object information following engagement with the object.

25. The method as claimed in claim 24, wherein the method further includes the step of providing a plurality of processing stations for processing objects, and a routing system for routing an object from the supply location to a selected object processing station capable of processing objects in the assigned class, wherein the selected object processing station processes the object according to the feature information regarding the objects.

26. The method as claimed in claim 24, wherein the method further includes the step of providing an evaluation system for evaluating the interaction of the object processing station with the object according to the processing parameters, wherein the evaluation system determines whether the interaction was successful or unsuccessful.

27. The method as claimed in claim 24 wherein the evaluation system determines the interaction success by evaluating object grasping, object movement, and object placement.

28. The method as claimed in claim 27 wherein the evaluation relies on sensor input.

29. The method as claimed in claim 27 wherein the evaluation relies on human input.

30. The system as claimed in claim 24 wherein the feature information is updated in response to the determination of the evaluation system.

31. The method as claimed in claim 24, wherein the feature information is developed associated with any of object stockkeeping unit (SKU), text description, product category, object manufacturer, object mass, object material, object shape, packaging details, object images, and object color.

32. The method as claimed in claim 24, wherein the feature information is developed associated with any of sensed information, object weight, object material, object shape, object opacity, object size, object volume, effector to be used, grip location, object movement limitations, object fragility, and object processing limitations.

33. A method of processing objects, said method comprising the steps of:

providing, at a supply location, one or more objects to be processed;

capturing identification data for the object;

querying a first data repository having a first set of object information;

determining that the identification data for the object is not included within the first set of object information;

identifying object feature data regarding the object;

querying a second data repository having a second set of object information, said second set of object information including feature information regarding a plurality of objects;

identifying associated object information within the second set of object information, said associated object information including learned feature data that closely matches the object feature data, said learned feature data being associated with a related object; and

engaging with a programmable motion device, the object using grasp and acquisition data regarding the related object.

34. The method as claimed in claim 33, wherein the method further includes the step of providing a plurality of processing stations for processing objects, and a routing system for routing an object from the supply location to a selected object processing station capable of processing objects in the assigned class, wherein the selected object processing station processes the object according to the feature information regarding the objects.

35. The method as claimed in claim 33, wherein the method further includes the step of providing an evaluation system for evaluating the interaction of the object processing station with the object according to the processing parameters, wherein the evaluation system determines whether the interaction was successful or unsuccessful.

36. The method as claimed in claim 33, wherein the feature information is developed associated with any of object stockkeeping unit (SKU), text description, product category, object manufacturer, object mass, object material, object shape, packaging details, object images, and object color.

37. The method as claimed in claim 33, wherein the feature information is developed associated with any of sensed information, object weight, object material, object shape, object opacity, object size, object volume, effector to be used, grip location, object movement limitations, object fragility, and object processing limitations.