Generating a trained robot path based on physical manipulation of the robot and based on training user interface input(s) associated with the physical manipulation

• US 9,925,662 B1
• Filed: 06/28/2015
• Issued: 03/27/2018
• Est. Priority Date: 06/28/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Jules, Anthony Sean, Jessen, Johan Ulrich Lewin
• Primary Examiner(s): Kiswanto, Nicholas
• Application Number:
  US14/752,966
• Time in Patent Office: 1,003 Days
• CPC Class Codes: B25J 9/16 Programme controls programme controls in general G05B19/00, e.g. numerical pogramme controls G05B19/18; recording or playback systems G05B19/42
• CPC Class Codes: G05B 19/423 Teaching successive positions by walk-through, i.e. the tool head or end effector being grasped and guided directly, with or without servo-assistance, to follow a path
• CPC Class Codes: G05B 2219/35488 Graphical user interface, labview
• CPC Class Codes: G05B 2219/36463 Manual switch to drive motor to wanted position, store, memorize position
• CPC Class Codes: G05B 2219/36492 Record position and orientation, posture of probe, tool
• CPC Class Codes: G05B 2219/36495 Recording position and other parameters, current, tool diameter, voltage
• CPC Class Codes: Y10S 901/04 Manual lead through

Abstract

Methods, apparatus, systems, and computer-readable media are provided for training a path of a robot by physically moving the robot, wherein the particular trained path and/or particular robot component movements to achieve the trained path are dependent on which of a plurality of available user interface inputs are selected for the training. The trained path defines a path to be traversed by a reference point of the robot, such as a path to be traversed by a reference point of an end effector of the robot. The particular robot component movements to achieve the trained path include, for example, the orientations of various robot components at each of a plurality of positions along the path, the velocity of various components at each of a plurality of positions along the path, etc.

Claims

1. A method of generating one or more trained paths for a robot, comprising:

identifying a first instance of a waypoint user interface input initiated by a user;

storing a first waypoint indication based on identifying the first instance, the first waypoint indication defining a first position of a reference point of the robot at the first instance;

identifying a second instance of the waypoint user interface input, the second instance initiated by the user after the first instance and initiated by the user following manipulation of the robot by the user to displace the reference point from the first position;

storing a second waypoint indication based on identifying the second instance, the second waypoint indication defining a second position of the reference point of the robot at the second instance;

generating a waypoint trained path segment for a trained path for the robot based on the first waypoint indication and the second waypoint indication, wherein generating the waypoint trained path segment comprises;

determining a first waypoint of the waypoint trained path segment based on the first waypoint indication,

determining a second waypoint of the waypoint trained path segment based on the second waypoint indication, and

determining an intermediate path of the waypoint trained path segment, the intermediate path being between the first waypoint and the second waypoint and being determined based on one or more optimization parameters;

identifying a special action user interface input initiated by the user;

generating a special action trained path segment for the trained path for the robot based on identifying the special action user interface input, the special action trained path segment conforming to a path traversed by the reference point in response to physical manipulation of the robot by the user during the special action user interface input;

wherein the waypoint trained path segment and the special action trained path segment are consecutive segments of the trained path.

2. The method of claim 1, further comprising:

generating special action orientations for the special action trained path segment, each of the special action orientations conforming to a respective orientation of the robot at a respective position of the path traversed by the reference point in response to manipulation of the robot by the user during the special action user interface input.

3. The method of claim 2, further comprising:

generating special action velocities for the special action trained path segment, each of the special action velocities conforming to a respective velocity of the reference point at a respective position of the path traversed by the reference point in response to manipulation of the robot by the user during the special action user interface input.

4. The method of claim 2, wherein generating the special action trained path segment and the special action orientations comprises:

generating a plurality of vectors that collectively define the special action trained path segment and the special action orientations, each of the vectors defining one or more values for each of a plurality of actuators of the robot.

5. The method of claim 1, further comprising generating intermediate path orientations for the intermediate path of the waypoint trained path segment, the intermediate path orientations generated based on one or more orientation optimization parameters and each of the intermediate path orientations conforming to a respective orientation of the robot at a respective position of the intermediate path.

6. The method of claim 1, further comprising generating special action orientations for the special action trained path segment, the special action orientations generated based on one or more orientation optimization parameters.

7. The method of claim 6, further comprising:

identifying an additional special action user interface input initiated by the user;

generating an additional special action trained path segment for the robot based on identifying the additional special action user interface input, the additional special action trained path segment conforming to an additional path traversed by the reference point in response to manipulation of the robot by the user during the additional special action user interface input; and

generating additional special action orientations for the additional special action trained path segment, each of the additional special action orientations conforming to a respective orientation of the robot at a respective position of the additional path traversed by the reference point in response to manipulation of the robot by the user.

8. The method of claim 1, wherein the waypoint user interface input initiated by the user is based on actuation of a user interface element in a first manner and wherein the special action user interface input initiated by the user is based on actuation of the user interface element in a second manner.

9. The method of claim 8, wherein the user interface element is on an exterior surface of an operational component of the robot.

10. The method of claim 8, wherein the first manner comprises touching the user interface element and ceasing touching of the user interface element within a threshold amount of time.

11. The method of claim 10, wherein the second manner comprises touching the user interface element and continuing touching the user interface element beyond the threshold amount of time.

12. The method of claim 11, wherein an initial touching of the user interface element indicates a start of the special action user interface input and wherein ceasing touching the user interface element indicates a stop of the special action user interface input.

13. The method of claim 1, wherein the waypoint user interface input initiated by the user comprises actuation of a first user interface element and wherein the special action user interface input initiated by the user comprises actuation of a second user interface element.

14. The method of claim 13, wherein the special action user interface input initiated by the user is based on the user touching the second user interface element and continuing touching the second user interface element, wherein an initial touching of the second user interface element indicates a start of the special action user interface input and wherein ceasing touching the second user interface element indicates a stop of the special action user interface input.

15. The method of claim 13, wherein the special action user interface input initiated by the user is based on the user touching the second user interface element and continuing touching the second user interface element, wherein the initial touching of the second user interface element indicates a start of the special action user interface input and wherein either of cessation of movement of the robot by the user or passage of a threshold amount of time indicates a stop of the special action user interface input.

16. The method of claim 1, further comprising:

traversing, by the robot, the reference point along the waypoint trained path segment and along the special action trained path segment.

17. The method of claim 1, further comprising:

providing the waypoint trained path segment and the special action trained path segment to a robot programming application for graphical display of the waypoint trained path segment and the special action trained path segment.

18. A system comprising:

a robot that includes one or more operational components that may be physically moved by a user in a training mode of the robot;

one or more user interface elements;

memory storing instructions;

one or more processors operable to execute the instructions stored in the memory to cause one or more of the processors to;

identify a first instance of a waypoint user interface input in response to a waypoint user actuation of at least one of the user interface elements by a user;

store a first waypoint indication based on identifying the first instance, the first waypoint indication defining a first position of a reference point of the robot at the first instance as sensed by the robot;

identify a second instance of the waypoint user interface input in response to the waypoint user actuation, the second instance occurring following manipulation of the robot by the user to displace the reference point from the first position;

store a second waypoint indication based on identifying the second instance, the second waypoint indication defining a second position of the reference point of the robot at the second instance as sensed by the robot;

generate a waypoint trained path segment for a trained path for the robot based on the first waypoint indication and the second waypoint indication;

identify a special action user interface input in response to a special action user actuation of at least one of the user interface elements by a user; and

generate a special action trained path segment for the trained path for the robot based on identifying the special action user interface input, the special action trained path segment conforming to a path traversed by the reference point in response to physical manipulation of the robot by the user during the special action user interface input;

wherein the waypoint trained path segment and the special action trained path segment are consecutive segments of the trained path.

19. The system of claim 18, wherein in executing the instructions stored in the memory one or more of the processors are further to:

generate special action orientations for the special action trained path segment, the special action orientations generated based on one or more orientation optimization parameters;

identify an additional special action user interface input initiated by the user;

generate an additional special action trained path segment for the robot based on identifying the additional special action user interface input, the additional special action trained path segment conforming to an additional path traversed by the reference point in response to manipulation of the robot by the user during the additional special action user interface input; and

generate additional special action orientations for the additional special action trained path segment, each of the additional special action orientations conforming to a respective orientation of the robot at a respective position of the additional path traversed by the reference point in response to manipulation of the robot by the user.

20. The system of claim 18, wherein the waypoint user actuation is actuation of a given user interface element in a first manner and wherein the special action user actuation is actuation of the given user interface element in a second manner.

21. At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by a computing system, cause the computing system to perform the following operations:

identifying a first instance of a waypoint user interface input initiated by a user;

storing a first waypoint indication based on identifying the first instance, the first waypoint indication defining a first position of a reference point of a robot at the first instance;

identifying a second instance of the waypoint user interface input, the second instance initiated by the user after the first instance and initiated by the user following manipulation of the robot by the user to displace the reference point from the first position;

storing a second waypoint indication based on identifying the second instance, the second waypoint indication defining a second position of the reference point of the robot at the second instance;

generating a waypoint trained path segment for a trained path for the robot based on the first waypoint indication and the second waypoint indication, wherein generating the waypoint trained path segment comprises;

determining a first waypoint of the waypoint trained path segment based on the first waypoint indication,

determining a second waypoint of the waypoint trained path segment based on the second waypoint indication, and

determining an intermediate path of the waypoint trained path segment, the intermediate path being between the first waypoint and the second waypoint and being determined based on one or more optimization parameters;

identifying a special action user interface input initiated by the user;

generating a special action trained path segment for the trained path for the robot based on identifying the special action user interface input, the special action trained path segment conforming to a path traversed by the reference point in response to physical manipulation of the robot by the user during the special action user interface input;

wherein the waypoint trained path segment and the special action trained path segment are consecutive segments of the trained path.