Generating a grasp affordance for an object based on a thermal image of the object that is captured following human manipulation of the object

• US 9,616,568 B1
• Filed: 08/25/2015
• Issued: 04/11/2017
• Est. Priority Date: 08/25/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Russell, Jared Stephen
• Primary Examiner(s): Moyer, Dale
• Application Number:
  US14/835,424
• Time in Patent Office: 595 Days
• CPC Class Codes: B25J 19/02 Sensing devices
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/37426 Detected with infrared sensor
• CPC Class Codes: G05B 2219/39546 Map human grasps to manipulator grasps
• CPC Class Codes: G05B 2219/40116 Learn by operator observation, symbiosis, show, watch
• CPC Class Codes: Y10S 901/03 Teaching system

Abstract

Methods, apparatus, and computer readable storage media related to utilizing a thermographic camera to capture at least one thermal image of an object following human manipulation of the object, and generating a grasp affordance for the object based on the temperatures indicated by the captured thermal image. The generated grasp affordance may be utilized, directly or indirectly, by one or more robots for determining grasping parameters for manipulating the object and/or other objects that are similar to the object.

Claims

1. A method, comprising:

detecting an object based on data from one or more sensors of a robot;

capturing, by a thermographic camera of the sensors of the robot, at least one thermal image of the object prior to expiration of a threshold amount of time following completion of manipulation of the object by a human;

capturing, by the thermographic camera, at least one additional thermal image of the object, the additional thermal image of the object captured either before the manipulation of the object by the human or after expiration of the threshold amount of time following the completion of the manipulation of the object by the human;

generating, by one or more processors, a grasp affordance for the object based on differences in temperatures indicated by the thermal image and the additional thermal image, the grasp affordance defining one or more portions of the object touched by the human during the manipulation of the object by the human; and

storing the grasp affordance for the object in a non-transitory computer readable storage medium for determining one or more grasping parameters for manipulating the object or other objects that are similar to the object.

2. The method of claim 1, wherein generating the grasp affordance for the object comprises:

determining the differences in the temperatures between the one or more portions in the thermal image and the one or more portions in the additional thermal image; and

defining the one or more portions of the object based on the differences in the temperatures satisfying a threshold.

3. The method of claim 2, wherein the grasp affordance for the object defines the one or more portions of the object touched by the human during the manipulation and defines additional portions of the object.

4. The method of claim 3, wherein the grasp affordance for the object comprises one or more geometric shapes that encompass the one or more portions and the additional portions.

5. The method of claim 1, further comprising:

assigning the grasp affordance to corresponding portions of the object in a non-thermographic image of the object, the non-thermographic image captured by a non-thermographic camera of the sensors of the robot;

wherein storing the grasp affordance for the object comprises storing the assignment of the grasp affordance to the corresponding portions of the object in the non-thermographic image of the object.

6. The method of claim 1, further comprising:

assigning the grasp affordance to corresponding portions of the object in a three dimensional model of the object;

wherein storing the grasp affordance for the object comprises storing the assignment of the grasp affordance to the corresponding portions of the object in the three dimensional model of the object.

7. The method of claim 1, further comprising:

determining an identifier of the object based on the data from the one or more of the sensors of the robot; and

assigning the grasp affordance to the identifier of the object.

8. The method of claim 1, further comprising:

determining, by the robot, an indication of the completion of the manipulation of the object by the human based on the data from the one or more sensors of the robot; and

selecting the thermal image for generating the grasp affordance based on determining the thermal image was captured following the indication of the completion of the manipulation of the object.

9. The method of claim 8, further comprising:

determining, by the robot, an indication of a start of the manipulation of the object by the human based on the data from the one or more sensors of the robot; and

selecting the additional thermal image for generating the grasp affordance based on determining the thermal image was captured prior to the indication of the start of the manipulation of the object.

10. The method of claim 1, further comprising:

generating grasping parameters for the object based on the grasp affordance; and

positioning a grasping end effector of the robot based on the grasping parameters.

11. The method of claim 1, further comprising:

generating a training example based on the grasp affordance; and

providing the training example for training of a convolutional neural network to determine grasping parameters for manipulating objects.

12. The method of claim 1, wherein detecting the object based on the data from the one or more of the sensors of the robot comprises detecting the object based on data from a non-thermographic camera of the sensors.

13. The method of claim 1, wherein detecting the object based on the data from the one or more of the sensors of the robot comprises detecting the object based on data from the thermographic camera.

14. A method, comprising:

capturing, by a thermographic camera, at least one thermal image of an object following manipulation of the object by a human;

generating, by one or more processors, a grasp affordance for the object based on one or more temperatures indicated by the thermal image, the grasp affordance defining one or more portions of the object touched by the human during the manipulation of the object by the human; and

storing the grasp affordance for the object in a non-transitory computer readable storage medium for determining one or more grasping parameters for one or more robots in manipulating the object or other objects that are similar to the object.

15. The method of claim 14, wherein generating the grasp affordance for the object comprises:

defining the one or more portions of the object based on the temperatures indicated in the one or more portions in the thermal image being at least a threshold temperature greater than the temperatures indicated in other portions of the thermal image.

16. The method of claim 15, wherein generating the grasp affordance for the object further comprises:

defining the one or more portions of the object based on the one or more portions being at least a threshold size.

17. The method of claim 14, further comprising:

determining, by the one or more processors, an indication of completion of the manipulation of the object; and

selecting, by the one or more processors, the thermal image for generating the grasp affordance based on determining the thermal image was captured following the indication of the completion of the manipulation of the object.

18. The method of claim 14, further comprising:

generating grasping parameters for the object based on the grasp affordance; and

positioning a grasping end effector of one of the robots based on the grasping parameters.

19. The method of claim 14, further comprising:

capturing, by the thermographic camera, at least one additional thermal image of the object, the additional thermal image of the object captured either before the manipulation of the object by the human or after expiration of a threshold amount of time following completion of the manipulation of the object by the human;

wherein generating the grasp affordance for the object based on the one or more temperatures indicated by the thermal image comprises;

determining differences between the one or more temperatures indicated by the thermal image and one or more temperatures indicated by the additional thermal image; and

defining the one or more portions of the object for the grasp affordance based on the differences in the temperatures for the one or more portions satisfying a threshold.

20. A system, comprising:

a thermographic camera capturing thermal images;

one or more processors executing instructions stored in a non-transitory computer readable storage medium, wherein execution of the instructions cause the one or more processors to;

receive at least one thermal image of the thermal images, the thermal image captured by the thermographic camera following completion of manipulation of the object by a human;

generate a grasp affordance for the object based on one or more temperatures indicated by the thermal image, the grasp affordance defining one or more portions of the object touched by the human during the manipulation of the object by the human; and

store the grasp affordance for the object in a non-transitory computer readable storage medium for determining one or more grasping parameters for one or more robots in manipulating the object or other objects that are similar to the object.

21. The system of claim 20, further comprising:

a robot, wherein the thermographic camera is mechanically coupled to the robot.