Validation of robotic item grasping

• US 9,669,543 B1
• Filed: 12/11/2015
• Issued: 06/06/2017
• Est. Priority Date: 12/11/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Stubbs, Andrew, Marchese, Andrew D., Krishnaswamy, Sarath, Ort, Moses Theodore, Toris, Russell Charles, Wachlin, Jacob Taylor, Longtine, John Gregory
• Primary Examiner(s): Sample, Jonathan L
• Application Number:
  US14/966,958
• Time in Patent Office: 543 Days
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: G05B 2219/39476 Orient hand relative to object
• CPC Class Codes: G05B 2219/40386 Search around teached point until operation has succes, correct program
• CPC Class Codes: Y10S 901/32 Servo-actuated
• CPC Class Codes: Y10S 901/46 Sensing device

Abstract

A grasp management system and corresponding method are described. In some examples, it is determined whether an end of arm tool of a robotic manipulator is capable of grasping an item in a first orientation and a second orientation. Information regarding success of the attempts may be retained in a database. A richness measure for each of the first orientation and the second orientation may be determined based upon a spatial variation of successful attempted grasps in the respective orientation.

Claims

1. A system, comprising:

at least one processor; and

a memory including instructions that, when executed by the at least one processor cause the system to, at least;

access information about an item;

access information about an end of arm tool of a robotic manipulator comprising a grasping function capable of grasping the item; and

determine a richness measure for multiple orientations of the item for a set of grasps for use by the end of arm tool, each of the grasps comprising utilization of the grasping function to successfully grasp the item via one or more contact points on the item, the richness measure representing a spatial variation of grasps of the set of grasps with respect to each orientation of the item, the richness measure for each orientation determined by at least;

determining whether the end of arm tool is capable of grasping the item using the grasping function at a first set of contact points located on the item when the item is in the orientation with respect to the end of arm tool;

determining a quality value of the grasp at the first set of contact points;

iteratively determining additional sets of contacts points in the orientation at which the end of arm tool is capable of grasping the item using the grasping function, and quality values of grasps at the additional sets of contact points; and

determining the richness measure for the orientation based at least in part upon the number of grasps in the orientation and the quality values of the grasps.

2. The system of claim 1, wherein the set of grasps comprises:

at least one successful learned grasp that was attempted in response to input from a human operating the robotic manipulator; and

at least one successful grasp that was attempted by the robotic manipulator without input from the human.

3. The system of claim 1, wherein determining whether the end of arm tool is capable of grasping the item using the grasping function comprises:

simulating, in a simulation environment, multiple attempts to grasp a simulated item using a simulated end of arm tool;

recording, in a data structure associated with the simulation environment and for each of the multiple attempts, whether the simulated end of arm tool successfully grasped the simulated item; and

determining, based at least in part on information in the data structure, a rate of success for each simulated grasp of the set of grasps.

4. The system of claim 1, wherein determining whether the end of arm tool is capable of grasping the item using the grasping function comprises:

attempting, in a testing environment, multiple attempts to grasp the item using the end of arm tool;

recording, in a data structure associated with the testing environment and for each of the multiple attempts, whether the end of arm tool successfully grasped the item; and

determining, based at least in part on information in the data structure, a rate of success for each grasp of the set of grasps.

5. The system of claim 1, wherein determining the quality value of the grasp at the first set of contact points is based at least in part on sensing information received from one or more sensors associated with at least one of the end of arm tool or the robotic manipulator.

6. A computer-implemented method, comprising:

accessing information about an item in a first item orientation relative to an end of arm tool of a robotic manipulator comprising a grasping function;

determining whether the end of arm tool is capable of grasping the item at any of a plurality of contact points located on the item using the grasping function when the item is in the first orientation;

maintaining information about first successful attempted grasps that represent that the end of arm tool successfully grasped the item in the first orientation;

accessing information about the item in a second item orientation relative to the end of arm tool of the robotic manipulator;

determining whether the end of arm tool is capable of grasping the item at any of the plurality of contact points located on the item using the grasping function when the item is in the second item orientation;

maintaining information about second successful attempted grasps that represent that the end of arm tool successfully grasped the item in the second orientation; and

determining a richness measure for each of the first orientation and the second orientation based upon a spatial variation of successful attempted grasps in the respective orientation.

7. The computer-implemented method of claim 6, wherein the first successful attempted grasps comprise a successful learned grasp that was attempted in response to input from a human operating the robotic manipulator.

8. The computer-implemented method of claim 6, wherein the first item orientation comprises the item oriented with a side facing up and the second item orientation comprises the item oriented with the side facing down.

9. The computer-implemented method of claim 6, wherein a bottom side of the item rests on a surface in the first item orientation, and the item is rotated about an axis extending about normal to the surface in the second item orientation.

10. The computer-implemented method of claim 6, wherein determining whether the end of arm tool is capable of grasping the item when the item is in the first orientation comprises, iteratively, for each contact point of the plurality of contact points:

instructing the robotic manipulator to cause the end of arm tool to pick up the item at each contact point of the plurality of contact points; and

recording in a data structure success or failure of the end of arm tool to pick up the item at each contact point of the plurality of contact points.

11. The computer-implemented method of claim 6, wherein the first successful attempted grasps and the second successful attempted grasps define a set of successful attempted grasps.

12. The computer-implemented method of claim 11, wherein the richness measure for each of the first orientation and the second orientation comprises first and second richness measures, the method further comprising:

expanding the set of successful attempted grasps to form an expanded set of successful attempted grasps by at least;

determining whether the end of arm tool is capable of grasping the item at any of the plurality of contact points located on the item using the grasping function when the end of arm tool is in a first grasping orientation relative to the item;

maintaining information about third successful attempted grasps that represent that the end of arm tool successfully grasped the item when the end of arm tool is in the first grasping orientation;

accessing information about the end of arm tool being in a second grasping orientation relative to the item;

determining whether the end of arm tool is capable of grasping the item at any of the plurality of contact points located on the item using the grasping function when the end of arm tool is in the second grasping orientation;

maintaining information about fourth successful attempted grasps that represent that the end of arm tool successfully grasped the item when the end of arm tool is in the second grasping orientation; and

determining third and fourth richness measures for each of the first grasping orientation and the second grasping orientation based upon a spatial variation of successful attempted grasps in the respective grasping orientation; and

determining, based on the first and second richness measures and the third and fourth richness measures, an overall richness measure corresponding to the expanded set of successful attempted grasps.

13. The computer-implemented method of claim 12, wherein the first grasping orientation comprises the end of arm tool approaching the item at an upper portion of the item, and the second grasping orientation comprises the end of arm tool approaching the item from a side portion of the item.

14. The computer-implemented method of claim 6, wherein at least one contact point of the plurality of contact points corresponds to a fiducial marker indicating a suggested grasping location for the end of arm tool to grasp the item.

15. The computer-implemented method of claim 6, wherein determining the richness measure for each of the first orientation and the second orientation is further based upon the spatial variation of successful attempted grasps as compared to a surface area of the item.

16. A computer-implemented method, comprising:

accessing information about a set of grasps for an end of arm tool of a robotic manipulator to grasp an item;

accessing information about a surface of the item when the item is in an orientation; and

creating a success surface area based on the orientation by at least;

when the item is in the orientation, utilizing a first grasp from the set of grasps, successfully picking up the item by grasping a first location on the surface of the item with the end of arm tool;

when the item is in the orientation, utilizing a second grasp from the set of grasps, successfully picking up the item by grasping a second location on the surface of the item with the end of arm tool; and

recording, in a data structure, the success surface area defined by a spatial variation between the first location and the second location.

17. The computer-implemented method of claim 16, wherein the orientation comprises a first stable orientation of the item previously identified as being stable.

18. The computer-implemented method of claim 16, wherein the success surface area is determined when the end of arm tool is in a first grasping orientation with respect to the item, the method further comprising:

accessing information indicating that the end of arm tool is in a second grasping orientation with respect to the item;

utilizing the first grasp from the set of grasps, successfully picking up the item by grasping the item at the first location when the end of arm tool is in the second grasping orientation;

utilizing the second grasp from the set of grasps, successfully picking up the item by grasping the item at the second location when the end of arm tool is in the second grasping orientation; and

updating the success surface area based at least in part on the end of arm tool successfully picking up the item at the first and second locations in the second grasping orientation.

19. The computer-implemented method of claim 16, wherein creating the success surface area comprises iteratively, utilizing each grasp from the set of grasps, successfully picking up the item by grasping various locations on the surface of the item with the end of arm tool, the success surface area defined by a spatial variation among the first location, the second location, and the various locations.

20. The computer-implemented method of claim 16, wherein in response to the success surface area failing to meet a threshold value, the method further comprising generating a notification that at least indicates the set of grasps as being incomplete.