Robotic training apparatus and methods

• US 9,384,443 B2
• Filed: 06/14/2013
• Issued: 07/05/2016
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Passot, Jean-Baptiste, Sinyavskiy, Oleg, Ponulak, Filip, Laurent, Patryk, Gabardos, Borja Ibarz, Izhikevich, Eugene
• Primary Examiner(s): Edwards, Jerrah
• Assistant Examiner(s): MOTT, ADAM R
• Application Number:
  US13/918,338
• Publication Number:
  US 20140371907A1
• Time in Patent Office: 1,117 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Apparatus and methods for training of robotic devices. Robotic devices may be trained by a user guiding the robot along target trajectory using an input signal. A robotic device may comprise an adaptive controller configured to generate control commands based on one or more of the user guidance, sensory input, and/or performance measure. Training may comprise a plurality of trials. During first trial, the user input may be sufficient to cause the robot to complete the trajectory. During subsequent trials, the user and the robot'"'"'s controller may collaborate so that user input may be reduced while the robot control may be increased. Individual contributions from the user and the robot controller during training may be may be inadequate (when used exclusively) to complete the task. Upon learning, user'"'"'s knowledge may be transferred to the robot'"'"'s controller to enable task execution in absence of subsequent inputs from the user.

Claims

1. A computerized controller apparatus configured to effectuate control of a robotic device, the controller apparatus comprising:

one or more processors configured to execute computer program modules to cause the one or more processors to;

(1) during a first training trial of a plurality of trials;

determine a first signal based on a characteristic of an input provided to the controller apparatus by the robotic device; and

cause the robotic device to perform a first action based on a first control signal, the first control signal being determined based on the first signal and a first user input, the first action being characterized by a first performance measure; and

(2) during a second training trial of the plurality of trials, the second training trial being subsequent to the first training trial;

determine a second signal based on the characteristic of the input and a first error, the first error being determined based on a proximity measure between the first action and a target action, the first error being greater than a third error corresponding to a third action performed by the robotic device responsive to a third control signal configured based solely on the first user input; and

cause the robotic device to perform a second action based on a second control signal, the second control signal being determined based on the second signal and a second user input, the second action being characterized by a second error, the second error being determined based on a proximity measure between the second action and the target action;

wherein;

the second error is smaller than the first error.

2. The apparatus of claim 1, wherein the computer program modules are further executable to cause the one or more processors to:

during a third training trial of the plurality of trials, the third training trial being subsequent to the second training trial;

determine a third signal based on the characteristic of the input and the second error; and

cause the robotic device to perform a fourth action based on a fourth control signal, the fourth control signal being determined based on the third signal absent a user input, the fourth action being characterized by a fourth error, the fourth error being determined based on a proximity measure between the fourth action and the target action, the fourth error being lower than the first error.

3. The apparatus of claim 1, wherein:

the robotic device comprises a motor configured to receive one or more operational directives capable of causing displacement of the motor; and

the first and second user inputs comprise the one or more operational directives configured in accordance with the target action.

4. The apparatus of claim 1, wherein:

the robotic device comprises a motor configured to receive one or more operational directives comprising an electrical signal characterized by one or more of amplitude, polarity, frequency, duration, or periodicity; and

the first and second user inputs comprise the one or more operational directives configured in accordance with the target action.

5. The apparatus of claim 1, wherein:

the plurality of trials comprises more than ten trials;

individual ones of the plurality of trials are spaced by time intervals that are at most ten percent of a duration of the plurality of trials; and

for a given iteration of a plurality of iterations, the computer program modules are further executable to cause the one or more processors to effectuate control of the robotic device based on a combination of the user input and a controller-generated signal configured based on the characteristic.

6. The apparatus of claim 1, wherein:

the first user input associated with the first trial is characterized by a first signal parameter, the first signal parameter being selected from the group consisting of signal amplitude, signal duration, signal periodicity, signal polarity, signal phase, and signal frequency; and

the second user input associated with the second trial is characterized by a second signal parameter, the first signal parameter being selected from the group consisting of signal amplitude, signal duration, signal periodicity, signal polarity, signal phase, and signal frequency, the second parameter being different from the first parameter.

7. The apparatus of claim 1, wherein the first trial and the second trial are characterized by one or both of a different trial duration or a different inter-trial interval.

8. A method of training a computerized robotic apparatus to perform a target action based on sensory context, the training being effectuated via a plurality of iterations, the method comprising:

during a first iteration;

causing the apparatus to generate a first control signal based on the sensory context;

causing the apparatus to execute a first action based on the first control signal and a user input received from a user, the user input being indicative of the target action; and

causing the apparatus to adjust a learning parameter based on a first performance value determined based on the first action and the target action; and

during a subsequent iteration;

causing the apparatus to generate a second control signal based on the sensory context and the adjusted learning parameter; and

causing the apparatus to execute a second action based on the second control signal, the second action being characterized by a second performance value;

wherein;

the plurality of iterations is configured to occur within a time interval comprising at least a duration;

the first iteration and the subsequent iteration are separated by the duration;

a provision of the user input during the subsequent iteration is configured to cause a controller to execute a third action characterized by a third performance value; and

the second action is configured to deviate less from the target action as compared to the third action.

9. The method of claim 8, wherein;

a performance measure is determined by the apparatus; and

the first performance value is lower than the second performance value.

10. The method of claim 8, wherein;

the adjustment of the learning parameter is configured based on a supervised learning process configured based on the sensory context and a combination of the first control signal and the user input; and

the third performance value is lower than the second performance value.

11. A robotic apparatus, comprising:

a platform comprising one or more controllable elements; and

a controller comprising one or more processors configured to execute computer program modules configured to operate individual ones of the one or more controllable elements, the computer program modules comprising;

a first logic module configured to receive a first input from a human;

a second logic module configured to analyze the first input and to cause the platform to execute a first action in accordance with the first input; and

a third logic module configured to receive a second input from the human subsequent to the receipt of the first input, the second input being configured to cause a corrected action by the platform, the corrected action being characterized by a lower deviation from a target action;

wherein;

the execution of the first action is based on the operation of the individual ones of the one or more controllable elements;

the target action corresponds to an operation of the platform based on the first input from the human;

the analysis of the first input comprises a determination of a deviation between the first action and the target action; and

the analysis of the first input is configured to cause a modification of a controller state in accordance with a learning process configured based on a performance measure.

12. The apparatus of claim 11, wherein:

the second logic module comprises a predictor sub-module configured to determine a predicted control output based on a characteristic of a sensory signal, the sensory signal being configured to convey information associated with one or both of an environment of the robotic apparatus and a platform state;

the first input and the second input are configured based on the sensory signal; and

the first action is configured based on the predicted control output.

13. The apparatus of claim 11, wherein:

the corrected action is effectuated based on a cooperative interaction with the human, the cooperative interaction being characterized by a plurality of iterations;

the first input corresponds to a first given iteration of the plurality of iterations; and

the second input corresponds to a second given iteration of the plurality of iterations, the second given iteration occurring subsequent to the first given iteration.

14. A robotic apparatus, comprising:

a platform comprising one or more controllable elements; and

a controller comprising one or more processors configured to execute computer program modules configured to operate individual ones of the one or more controllable elements, the computer program modules comprising;

a first logic module configured to receive a first input from a human;

a second logic module configured to analyze the first input and to cause the platform to execute a first action in accordance with the first input; and

a third logic module configured to receive a second input from the human subsequent to the receipt of the first input, the second input being configured to cause a corrected action by the platform, the corrected action being characterized by a lower deviation from a target action;

wherein;

the execution of the first action is based on the operation of the individual ones of the one or more controllable elements;

the second logic module comprises a predictor sub-module configured to determine a predicted control output based on a characteristic of a sensory signal, the sensory signal being configured to convey information associated with one or both of an environment of the robotic apparatus and a platform state;

the first input and the second input are configured based on the sensory signal; and

the first action is configured based on the predicted control output.

15. The apparatus of claim 14, wherein:

the predictor sub-module is configured to provide a table configured to store a plurality of human inputs, a plurality of sensory signal characteristics, and a plurality of predicted control outputs; and

the analysis of the first input comprises a selection of a given predicted control output based on a match between a given sensory signal characteristic and an individual one of the plurality of sensory signal characteristics.

16. The apparatus of claim 15, wherein the given predicted control is configured based on a search of the table, the search being configured based on the first input.

17. The apparatus of claim 14, wherein:

the third logic module comprises a combiner sub-module configured to determine a combined output based on the predicted control output and the first input, the combined output being characterized by a transform function configured to combine the predicted control output and the first input via one or more operations including an additive operation.

18. The apparatus of claim 17, wherein the transform function is configured to combine the predicted control output and the first input via the one or more operations including a union operation.

19. The apparatus of claim 17, wherein:

a learning process comprises a supervised learning process configured based on the combined output;

the learning process is configured to be updated at time intervals; and

modification of the controller is based on an error measure between (i) the predicted control output generated at a first time instance and (ii) the first input determined at second time instance subsequent to the first time instance, the first time instance and the second time instance being separated by one of the time intervals.

20. A robotic apparatus, comprising:

a platform comprising one or more controllable elements; and

a controller comprising one or more processors configured to execute computer program modules configured to operate individual ones of the one or more controllable elements, the computer program modules comprising;

a first logic module configured to receive a first input from a human;

a second logic module configured to analyze the first input and to cause the platform to execute a first action in accordance with the first input; and

a third logic module configured to receive a second input from the human subsequent to the receipt of the first input, the second input being configured to cause a corrected action by the platform, the corrected action being characterized by a lower deviation from a target action;

wherein;

the execution of the first action is based on the operation of the individual ones of the one or more controllable elements;

the corrected action is effectuated based on a cooperative interaction with the human, the cooperative interaction being characterized by a plurality of iterations;

the first input corresponds to a first given iteration of the plurality of iterations; and

the second input corresponds to a second given iteration of the plurality of iterations, the second given iteration occurring subsequent to the first given iteration.

21. The apparatus of claim 20, wherein:

the second logic module comprises a predictor sub-module configured to determine a plurality of predicted control outputs based on a characteristic of a sensory signal and a given user input;

the sensory signal is configured to convey information about one or both of an environment of the robotic apparatus and a platform state;

the first input and the second input are configured based on the sensory signal;

the first action is configured based on a first predicted control output of the plurality of predicted control outputs, the first predicted control output corresponding to the first input;

the corrected action is configured based on a second predicted control output of the plurality of predicted control outputs, the second predicted control output corresponding to the second input;

the corrected action is characterized by an improved performance as compared to the first action and the target action;

the improved performance is quantified based on a lower deviation of the corrected action from the target action compared to a deviation between the first action and the target action; and

the target action is based solely on a training input by the human absent the first or the second predicted control output.

22. The apparatus of claim 20, wherein:

the target action corresponds to an operation of the platform based on the first input from the human;

the analysis of the first input comprises a determination of a deviation between the first action and the target action; and

the analysis of the first input is configured to cause a modification of a controller state in accordance with a learning process configured based on a performance measure.