Generating robotic grasping instructions for inventory items

• US 9,492,923 B2
• Filed: 12/16/2014
• Issued: 11/15/2016
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Wellman, Parris S., Verminski, Matthew David, Stubbs, Andrew, Shydo, Robert Michael Jr., Claretti, Ennio, Aronchik, Boris, Longtine, John Gregory
• Primary Examiner(s): Tran, Dalena
• Application Number:
  US14/572,420
• Publication Number:
  US 20160167228A1
• Time in Patent Office: 700 Days
• Field of Search: 700/245, 700/250, 700/224, 700/257, 700/259, 700213-216, 901 1- 2, 901/46, 901/47, 198/468.2, 318/568.12, 318/568.13, 318/568.16
• US Class Current: 1/1
• CPC Class Codes: B25J 9/0084 comprising a plurality of manipulators
• CPC Class Codes: B25J 9/1602 characterised by the control system, structure, architecture
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1669 characterised by special application, e.g. multi-arm co-operation, assembly, grasping
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 1/0492 with cars adapted to travel in storage aisles
• CPC Class Codes: B65G 1/10 with relatively movable racks to facilitate insertion or removal of articles cabinets with means for moving compartments up and down A47B51/00; cabinet system, e.g. consisting of cabinets arranged in a row with means to open or close passages between adjacent cabinets A47B53/02
• CPC Class Codes: B65G 1/1378 the orders being assembled on fixed commissioning areas remote from the storage areas
• CPC Class Codes: G05B 2219/39468 Changeable hand, tool, code carrier, detector
• CPC Class Codes: G05B 2219/39473 Autonomous grasping, find, approach, grasp object, sensory motor coordination
• CPC Class Codes: G05B 2219/39536 Planning of hand motion, grasping
• CPC Class Codes: G05B 2219/39543 Recognize object and plan hand shapes in grasping movements
• CPC Class Codes: G05B 2219/39546 Map human grasps to manipulator grasps
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: Y10S 901/03 Teaching system
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement
• CPC Class Codes: Y10S 901/30 End effector
• CPC Class Codes: Y10S 901/31 Gripping jaw
• CPC Class Codes: Y10S 901/47 : Optical

Abstract

Robotic arms may be utilized to grasp inventory items within an inventory system. Information about an inventory item to be grasped can be detected and used to determine a grasping strategy in conjunction with information from a database. Instructions for grasping an inventory item can be generated based on the detected information and the database.

Claims

1. A method comprising, under the control of one or more computer systems configured with executable instructions:

for each inventory item of a plurality of inventory items;

determining an identity of the inventory item to be grasped from among identities previously stored for inventory items;

accessing, based on the determined identity of the inventory item, a record from an item database including characteristics of the inventory item;

accessing, based at least in part on the characteristics of the inventory item, grasping strategies from an item grasping database; and

responsive to the inventory item not having a grasping strategy associated therewith;

generating a grasping strategy for the inventory item based at least in part on at least one of (i) characteristics of the inventory item that are at least one of stored in the record or detected by one or more sensors;

(ii) input from a human operator received through a human-based grasping strategy module for grasping the inventory item;

(iii) grasping strategies for items with characteristics similar to the characteristics of the inventory item;

(iv) physical constraints associated with a location in which the inventory item is to be grasped or placed;

or (v) information about a success of grasping strategies that have been previously employed;

instructing a robotic manipulator to perform the generated grasping strategy for the inventory item; and

storing the generated grasping strategy in association with the characteristics of the inventory item in the item grasping database.

2. The method of claim 1, further comprising:

receiving evaluation information about a success of the generated grasping strategy; and

updating the item grasping database based on the evaluation information.

3. The method of claim 2, wherein updating the item grasping database based on the evaluation information comprises, responsive to the evaluation information indicating that the generated grasping strategy was successful, designating the generated grasping strategy stored in association with the characteristics of the inventory item as the designated grasping strategy for the inventory item.

4. The method of claim 2, wherein updating the item grasping database based on the evaluation information comprises, responsive to the evaluation information indicating that the generated grasping strategy was unsuccessful, identifying the generated grasping strategy stored in association with the characteristics of the inventory item as an ineffective grasping strategy.

5. The method of claim 1, wherein generating a grasping strategy for the inventory item comprises generating a grasping strategy that is different from grasping strategies that are associated with the characteristics of the inventory item in the item grasping database as ineffective grasping strategies.

6. A non-transitory computer-readable storage medium having stored therein instructions that, when executed by one or more processors of a computer system, cause the computer system to perform operations comprising at least:

determining an identity of an inventory item to be grasped from a grasping environment;

accessing, based on the determined identity of the inventory item, a record in an item database about the inventory item, the record including information about one or more geometric characteristics of the inventory item;

receiving, from a sensor, one or more detected attributes of the inventory item in the grasping environment;

determining, based at least in part on the one or more geometric characteristics and the one or more detected attributes, an orientation of the inventory item in the grasping environment;

generating a grasping strategy using a robotic manipulator for the inventory item based at least in part on the determined orientation of the inventory item in the grasping environment, wherein generating the grasping strategy includes selecting an end effector from among different types of available end effectors; and

storing the generated grasping strategy in association with at least one of the one or more geometric characteristics, the one or more detected attributes, or the determined orientation of the inventory item in an item grasping database.

7. The non-transitory computer-readable storage medium of claim 6, wherein generating a grasping strategy for the inventory item based at least in part on the determined orientation of the inventory item in the grasping environment comprises:

instructing an alignment of the end effector based on the determined orientation of inventory item.

8. The non-transitory computer-readable storage medium of claim 6, wherein the end effector comprises at least one of a mechanical end effector, an electromechanical end effector, a vacuum end effector, an electro-adhesion end effector, a magnetic end effector, or a soft robotic end effector.

9. The non-transitory computer-readable storage medium of claim 6, wherein selecting an end effector to use in the grasping strategy comprises selecting at least one of (i) a type of end effector, (ii) an intensity at which the end effector is to be operated, or (iii) an orientation at which the robotic manipulator is to approach the inventory item;

and

wherein the selection is based at least in part on at least one of;

(i) one or more of the one or more geometric characteristics of the inventory item in the record (ii), one or more of the one or more detected attributes of the inventory item, or (iii) the determined orientation of inventory item.

10. The non-transitory computer-readable storage medium of claim 6, wherein generating a grasping strategy for the inventory item comprises generating the grasping strategy based at least in part on physical constraints associated with the grasping environment.

11. The non-transitory computer-readable storage medium of claim 6, wherein generating a grasping strategy for the inventory item comprises generating the grasping strategy based at least in part on information about a success of grasping strategies that have been previously employed.

12. The non-transitory computer-readable storage medium of claim 6, wherein generating a grasping strategy for the inventory item comprises generating the grasping strategy based at least in part on grasping strategies for items with at least one of:

(i) geometric characteristics similar to the one or more geometric characteristics of the inventory item, (ii) attributes similar to the one or more detected attributes of the inventory item, or (iii) orientations similar to the determined orientation of the inventory item.

13. The non-transitory computer-readable storage medium of claim 6, wherein generating a grasping strategy for the inventory item comprises generating the grasping strategy based at least in part on observation information from one or more sensors observing a human perform a grasping operation.

14. The non-transitory computer-readable storage medium of claim 6, wherein generating a grasping strategy for the inventory item comprises generating the grasping strategy based on at least one of instructions received via a user interface or actions observed in a virtual environment.

15. A non-transitory computer-readable storage medium having stored therein instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:

receive attribute information about an inventory item, the attribute information including at least one of an identity of the inventory item or information about physical characteristics of the inventory item;

receive observation information from one or more sensors observing a hand or other appendage of a human perform a grasping operation for the inventory item;

generate, based at least in part on the observation information, a grasping strategy for grasping the inventory item using a robotic manipulator; and

store the generated grasping strategy in a record in an item grasping database in association with the attribute information.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions, when executed by one or more processors of a computer system, further cause the computer system to at least:

instruct the robotic manipulator to execute the generated grasping strategy to grasp a target inventory item in response to receiving an indication that the target inventory item comprises attribute information in common with the attribute information of the inventory item.

17. The non-transitory computer-readable storage medium of claim 15, wherein the observation information includes information about an orientation at which the hand or other appendage of the human performs the grasping operation for the inventory item.

18. The non-transitory computer-readable storage medium of claim 15, wherein the observation information includes information about a pressure, shear force, or other physical characteristic with which the hand or other appendage of the human performs the grasping operation for the inventory item.

19. The non-transitory computer-readable storage medium of claim 15, wherein receiving observation information from one or more sensors observing a hand or other appendage of the human perform a grasping operation for the inventory item comprises receiving the observation information from the one or more sensors on a glove worn by the human.

20. The non-transitory computer-readable storage medium of claim 15, wherein the instructions, when executed by one or more processors of a computer system, further cause the computer system to at least provide real-time feedback and control of the robotic manipulator as the human performs the grasping operation.