METHOD AND SYSTEM FOR TRAINING A ROBOT USING HUMAN-ASSISTED TASK DEMONSTRATION

• US 20130245824A1
• Filed: 03/15/2012
• Published: 09/19/2013
• Est. Priority Date: 03/15/2012
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations Incorporated (General Motors Company)
• Inventors: Payton, David W., Barajas, Leandro G., Martinson, Eric, Uhlenbrock, Ryan M.
• Granted Patent: US 8,843,236 B2
• Time in Patent Office: Days
• US Class Current: 700/253
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: G05B 2219/40512 Real time path planning, trajectory generation

Abstract

A method for training a robot to execute a robotic task in a work environment includes moving the robot across its configuration space through multiple states of the task and recording motor schema describing a sequence of behavior of the robot. Sensory data describing performance and state values of the robot is recorded while moving the robot. The method includes detecting perceptual features of objects located in the environment, assigning virtual deictic markers to the detected perceptual features, and using the assigned markers and the recorded motor schema to subsequently control the robot in an automated execution of another robotic task. Markers may be combined to produce a generalized marker. A system includes the robot, a sensor array for detecting the performance and state values, a perceptual sensor for imaging objects in the environment, and an electronic control unit that executes the present method.

Claims

1. A method for training a robot to execute a robotic task in a work environment, the method comprising:

moving the robot across its configuration space through multiple states of the robotic task to thereby demonstrate the robotic task to the robot, wherein the configuration space is the set of all possible configurations for the robot;

recording motor schema describing a sequence of behavior of the robot;

recording sensory data describing performance and state values of the robot while moving the robot across its configuration space;

detecting perceptual features of objects located in the environment;

assigning virtual deictic markers to the detected perceptual features; and

using the assigned virtual deictic markers and the recorded motor schema to subsequently control the robot in an automated execution of another robotic task.

2. The method of claim 1, wherein moving the robot across its configuration space includes moving at least one of a robot arm and a robot manipulator attached to the robot arm.

3. The method of claim 2, wherein the manipulator includes a plurality of robotic fingers, and wherein moving the robot across its configuration space includes moving the robotic fingers.

4. The method of claim 1, wherein moving the robot across its configuration space includes manually backdriving at least one of a robotic arm and a manipulator of the robot.

5. The method of claim 1, wherein the robot is in communication with an input device, and wherein moving the robot across its configuration space includes commanding the movement via the input device.

6. The method of claim 1, wherein the input device is one of a joystick and a three-dimensional mouse.

7. The method of claim 1, wherein recording sensory data includes recording a force and a torque applied by the manipulator to an object grasped by the manipulator during demonstration of the task.

8. The method of claim 1, wherein detecting perceptual features of an object located in the environment includes using a camera to visually detect the perceptual features.

9. The method of claim 1, wherein using a camera to visually detect the perceptual features includes visually detecting a surface feature of the object.

10. The method of claim 9, wherein the camera is a three-dimensional stereo camera.

11. The method of claim 1, further comprising:

combining individual deictic markers to produce a generalized marker that maintains a record of perceptual features that are common between the combined individual markers, along with rotational and translational offsets required for the individual markers to match.

12. A method for training a robot to execute a robotic task in a work environment, the method comprising:

capturing data sequences of positions of a manipulator of the robot from user-controlled movements of the robot, wherein the user-controlled movements move the robot across its configuration space, and wherein the configuration space is the set of all possible configurations for the robot;

extracting data segments from the captured data sequences that represent actions of the robot during execution of a robotic skill;

detecting objects in the work environment of the robot;

assigning a virtual deictic marker to at least some of the detected objects to thereby associate an observed object'"'"'s spatial orientation in the work environment with movements performed by the robot relative to the object;

combining the deictic markers to produce a generalized marker, wherein the generalized marker maintains a record of visual features that are common between the deictic markers along with rotational and translational offsets required for the markers to match each other; and

using the generalized marker and the recorded motor schema to subsequently control the robot in an automated execution of another robotic task.

13. The method of claim 12, wherein the user-controlled movement includes manually guiding a robotic arm of the robot through the task.

14. The method of claim 13, wherein capturing data sequences of positions of a manipulator includes recording sensory information as the arm is manually guided through the task.

15. The method of claim 12, further comprising:

using models of known objects to identify orientations of objects in the environment having similar feature arrangements.

16. The method of claim 12, further comprising:

dynamically assigning the markers to detected perceptual features.

17. A system comprising:

a robot having an arm and a manipulator connected to the arm;

a sensor array which measures sensory data describing performance and state values of the robot;

a perceptual sensor which collects images of objects located in the environment; and

an electronic control unit (ECU) in communication with the robot, the sensor array, and the perceptual sensors, and which includes recorded motor schema describing a sequence of behavior of the robot, wherein the ECU is configured to;

record the sensory data when the arm and the manipulator are moved across the configuration space of the robot by a human operator through multiple states of a robotic task;

detect perceptual features in the collected images from the perceptual sensor;

assign virtual deictic markers to the detected perceptual features; and

use the assigned virtual deictic markers and the recorded motor schema to control the robot in an automated execution of another robotic task.

18. The system of claim 17, wherein the ECU is further configured to:

selectively combine individual deictic markers to produce a generalized marker that maintains a record of perceptual features that are common between the combined individual markers, along with rotational and translational offsets required for the individual markers to match.

19. The system of claim 17, wherein the ECU is configured to capture data sequences of positions of the manipulator from operator-controlled movements of the robot.

20. The system of claim 17, further comprising:

an input device in communication with the ECU, wherein the arm and the manipulator are moved by the human operator during recording of the sensory data using the input device.