Deep machine learning methods and apparatus for robotic grasping

• US 10,639,792 B2
• Filed: 01/26/2018
• Issued: 05/05/2020
• Est. Priority Date: 03/03/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Vijayanarasimhan, Sudheendra, Jang, Eric, Pastor Sampedro, Peter, Levine, Sergey
• Primary Examiner(s): Oh, Harry Y
• Application Number:
  US15/881,189
• Publication Number:
  US 20180147723A1
• Time in Patent Office: 830 Days
• Field of Search: 700250
• CPC Class Codes: B25J 9/1612 characterised by the hand, wrist, grip control
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 13/027 using neural networks only
• CPC Class Codes: G05B 19/18 Numerical control [NC], i.e. automatically operating machines, in particular machine tools, e.g. in a manufacturing environment, so as to execute positioning, movement or co-ordinated operations by means of programme data in numerical form G05B19/418 takes precedence
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: Y10S 901/36 Actuating means

Abstract

Deep machine learning methods and apparatus related to manipulation of an object by an end effector of a robot. Some implementations relate to training a semantic grasping model to predict a measure that indicates whether motion data for an end effector of a robot will result in a successful grasp of an object; and to predict an additional measure that indicates whether the object has desired semantic feature(s). Some implementations are directed to utilization of the trained semantic grasping model to servo a grasping end effector of a robot to achieve a successful grasp of an object having desired semantic feature(s).

Claims

1. A system, comprising:

a vision sensor viewing an environment of a robot;

a semantic grasping model stored in one or more non-transitory computer readable media;

at least one processor configured to;

identify a current image captured by the vision sensor;

generate, over a portion of the semantic grasping model based on application of the current image to the portion;

a measure of successful grasp, by a grasping end effector of the robot, of an object captured in the current image, wherein the measure of successful grasp indicates, directly or indirectly, a probability, and

spatial transformation parameters that indicate a location;

generate a spatial transformation, of the current image or of an additional image captured by the vision sensor, based on the spatial transformation parameters;

apply the spatial transformation as input to an additional portion of the semantic grasping model, wherein the additional portion is a deep neural network;

generate, over the additional portion and based on the spatial transformation, an additional measure that indicates whether a desired object semantic feature is present in the spatial transformation;

generate an end effector command based on the measure and the additional measure; and

provide the end effector command to one or more actuators of the robot.

2. The system of claim 1, wherein the desired object semantic feature defines an object classification.

3. The system of claim 1, further comprising a user interface input device, and wherein the at least one processor is further configured to:

receive user interface input from the user interface input device; and

identify the desired object semantic feature based on the user interface input.

4. The system of claim 3, wherein the user interface input device is a microphone.

5. The system of claim 1, wherein the spatial transformation is of the current image.

6. The system of claim 5, wherein the spatial transformation crops out a portion of the current image.

7. The system of claim 1, wherein the end effector command is a grasp command and wherein in generating the end effector command that is the grasp command, the at least one processor is configured to generate the grasp command based on:

the measure of successful grasp satisfying one or more criteria; and

the additional measure indicating that the desired object semantic feature is present in the spatial transformation.

8. The system of claim 1, wherein in generating the measure of successful grasp and the spatial transformation parameters, the at least one processor is further to:

generate the measure of successful grasp and the spatial transformation parameters based on applying a candidate end effector motion vector to the portion of the semantic grasping model.

9. The system of claim 8, wherein the image is applied to an initial layer of the portion and the candidate end effector motion vector is applied to an additional layer of the portion.

10. A method implemented by one or more processors, comprising:

identifying a desired object semantic feature for a grasp attempt;

generating a candidate end effector motion vector defining motion to move a grasping end effector of a robot from a current pose to an additional pose;

identifying a current image captured by a vision sensor associated with the robot, the current image capturing the grasping end effector and an object in an environment of the robot;

applying the current image and the candidate end effector motion vector as input to a trained semantic grasping model;

generating, based on processing of the current image and the candidate end effector motion vector using the trained semantic grasping model;

a measure of successful grasp of the object with application of the motion, and

an additional measure that indicates whether the object has the desired object semantic feature;

generating a grasp command based on determining that the measure of successful grasp satisfies one or more criteria and that the additional measure indicates that the object has the desired object semantic feature; and

providing the grasp command to one or more actuators of the robot to cause the end effector to attempt a grasp of the object.

11. The method of claim 10, wherein the desired object semantic feature defines an object classification.

12. The method of claim 10, further comprising:

receiving user interface input from a user interface input device;

wherein identifying the desired object semantic feature is based on the user interface input.