ROBOTIC TRAINING APPARATUS AND METHODS

• US 20140371907A1
• Filed: 06/14/2013
• Published: 12/18/2014
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Passot, Jean-Baptiste, Sinyavskiy, Oleg, Ponulak, Filip, Laurent, Patryk, Gabardos, Borja Ibarz, Izhikevich, Eugene
• Granted Patent: US 9,384,443 B2
• Time in Patent Office: Days
• US Class Current: 700/257
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: B25J 9/163 learning, adaptive, model based, rule based expert control
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G06N 3/008 based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
• CPC Class Codes: G06N 3/049 Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Apparatus and methods for training of robotic devices. Robotic devices may be trained by a user guiding the robot along target trajectory using an input signal. A robotic device may comprise an adaptive controller configured to generate control commands based on one or more of the user guidance, sensory input, and/or performance measure. Training may comprise a plurality of trials. During first trial, the user input may be sufficient to cause the robot to complete the trajectory. During subsequent trials, the user and the robot'"'"'s controller may collaborate so that user input may be reduced while the robot control may be increased. Individual contributions from the user and the robot controller during training may be may be inadequate (when used exclusively) to complete the task. Upon learning, user'"'"'s knowledge may be transferred to the robot'"'"'s controller to enable task execution in absence of subsequent inputs from the user

Claims

1. A computerized controller apparatus configured to effectuate control of a robotic device, the apparatus comprising:

one or more processors configured to execute computer program modules to cause one or more processors to;

(1) during a first training trial of a plurality of trials;

determine a first signal based on a characteristic of an input provided to the controller apparatus by the robotic device; and

cause the robotic device to perform a first action based on a first control signal, the first control signal being determined based on the first signal and a first user input, the first action being characterized by a first performance measure; and

(2) during a second training trial of the plurality of training trials, the second training train being subsequent to the first training trial;

determine a second signal based on the characteristic of the input and a first error, the first error being determined based on a proximity measure between the first action and a target action; and

cause the robotic device to perform a second action based on a second control signal, the second control signal being determined based on the second signal and a second user input, the second action being characterized by a second error, the second error being determined based on a proximity measure between the second action and the target action;

wherein;

the second error is smaller than the first error; and

the first error is greater than a third error corresponding to a third action performed by the robotic device responsive to a third control signal configured based solely on the first user input.

2. The apparatus of claim 1, wherein the computer program modules are further executable to cause one or more processors to:

during a third training trial of the plurality of trials, the third training trial being subsequent to the second training trial;

determine a third signal based on the characteristic of the input and the second error; and

cause the robotic device to perform a fourth action based on a fourth control signal, the fourth control signal being determined based on the third signal but absent user input, the fourth action being characterized by a fourth error, the fourth error being determined based on a proximity measure between the fourth action and the target action, the fourth error being lower than the first error.

3. The apparatus of claim 1, wherein:

the robotic device comprises a motor configured to receive an operational directive capable of causing displacement of the motor; and

the user input comprises one or more operational directives configured in accordance with the target action.

4. The apparatus of claim 1, wherein:

the robotic device comprises a motor configured to receive an operational directive comprising an electrical signal characterized by one or more of amplitude, polarity, frequency, duration, or periodicity; and

the user input comprises one or more operational directives configured in accordance with the target action.

5. The apparatus of claim 1, wherein:

the plurality of trials comprises more than ten trials;

individual ones of the plurality of trials are spaced by time intervals that are at most ten percent of duration of the plurality of trials; and

for a given iteration of the plurality of iterations, the computer program modules are further executable to cause one or more processors to effectuate the control of the robotic device based on a combination of the user input and a controller-generated signal configured based on the characteristic.

6. The apparatus of claim 1, wherein:

the first user input associated with the first trial is characterized by a first signal parameter, the first signal parameter being selected from the group consisting of signal amplitude, signal duration, signal periodicity, signal polarity, signal phase, and signal frequency; and

the second user input associated with the second trial is characterized by a second signal parameter, the first signal parameter being selected from the group consisting of signal amplitude, signal duration, signal periodicity, signal polarity, signal phase, and signal frequency, the second parameter being different from the first parameter.

7. The apparatus of claim 1, wherein the first trial and the second trial are characterized by one or both of a different trial duration or a different inter-trial interval.

8. A robotic apparatus, comprising:

a platform comprising one or more controllable elements; and

a controller comprising one or more processors configured to execute computer program modules configured to operate individual ones of the one or more controllable elements, the computer program modules comprising;

a first logic module configured to receive a first input from a human;

a second logic module configured to analyze the first input and to cause the platform to execute an action in accordance with the first input; and

a third logic module configured to receive a second input from the human subsequent to receipt of the first input, the second input being configured to cause a corrected action by the platform, the corrected action being characterized by lower deviation from a target action;

wherein the action execution is based on operation of individual ones of the one or more controllable elements.

9. The apparatus of claim 8, wherein:

the target action corresponds to operation of the platform based on teaching input from the human;

the analysis of the first input comprises determining a deviation between the first action and the target action data; and

the analysis is configured to cause modification of a controller state in accordance with a learning process configured based on the performance measure.

10. The apparatus of claim 8, wherein:

the second logic module comprises a predictor sub-module configured to determine a predicted control output based on a characteristic of sensory signal, the sensory signal conveying information associated with one or both of an environment of the robotic apparatus and a platform state;

the first input and the second input are configured based on the sensory signal; and

the action is configured based on the predicted control output.

11. The apparatus of claim 10, wherein:

the predictor sub-module is configured to provide a table configured to store a plurality of human inputs, a plurality of sensory signal characteristics, and a plurality of predicted control outputs and;

the analysis comprises a selection of a given predicted control output based on a match between a given characteristic of sensory signal and an individual one of the plurality of sensory signal characteristics.

12. The apparatus of claim 11, wherein the given predicted control is configured based on a search of the table, the search being configured based on the user input.

13. The apparatus of claim 10, wherein:

the third logic module comprises a combiner sub-module configured to determine a combined output based on the predicted control output and first user input, the combined output being characterized by a transform function configured to combine the predicted control output and the first signal via one or more operations including an additive operation.

14. The apparatus of claim 13, wherein the transform function is configured to combine the predicted control output and the control output via one or more operations including a union operation.

15. The apparatus of claim 13, wherein:

the learning process comprises a supervised learning process configured based on the combined output;

the learning process is configured to be updated at time intervals; and

the modification of the controller is based on an error measure between (i) the predicted control output generated at a first time instance and (ii) the first input determined at second time instance subsequent to the first time instance, the first time instance and the second time instance separated by one of the time intervals.

16. The apparatus of claim 8, wherein:

the corrected action is effectuated based on a cooperative interaction with the human, the cooperative interaction being characterized by a plurality of iterations;

the first input corresponds to a first given iteration of the plurality of iterations;

the second input corresponds to a second given iteration of the plurality of iterations, the second given iteration occurring subsequent to the first given iteration.

17. The apparatus of claim 16, wherein:

the second logic module comprises a predictor sub-module configured to determine a plurality of predicted control outputs based on a characteristic of a sensory signal and a given user input;

the sensory signal is configured to convey information about one or both of an environment of the robotic apparatus and a platform state;

the first input and the second input are configured based on the sensory signal;

the action is configured based on a first predicted control output of the plurality of predicted control outputs, the first predicted control output corresponding to the first input;

the corrected action is configured based on second predicted control output of the plurality of predicted control outputs, the second predicted control output corresponding to the second input;

the corrected action being characterized by improved performance as compared to the action and a target action;

the improved performance being quantified based on a lower deviation of the corrected action from the target action compared to deviation between the action and the target action; and

the target action being based solely on a training input by the human absent predicted control output.

18. A computerized robotic controller apparatus, comprising:

one or more processors configured to execute computer program modules to cause one or more processors to;

for a given iteration of a plurality of iterations;

generate a first signal based on a sensory context;

receive a second signal from a user associated with a target action configured based on the sensory context; and

utilize the first signal and the second signal to perform an action;

wherein;

the plurality of iterations is configured to occur within a time interval characterized by a duration;

a subsequent iteration of the plurality of iterations is configured to cause the action to be closer to the target action; and

upon expiration of the duration, the first signal is configured to perform the action absent the second signal.

19. The apparatus of claim 18, wherein:

individual ones of the plurality of iterations are spaced by time intervals that are at ten percent of the duration or less; and

the plurality of iterations include more than seven iterations.

20. A method of training a computerized robotic apparatus to perform a target action based on sensory context, the training being effectuated via a plurality of iterations, the method comprising:

during a first iteration;

causing the apparatus to generate a first control signal based on the sensory context;

causing the apparatus to execute a first action based on the first control signal and a user input received from a user, the user input being indicative of the target action; and

causing the apparatus to adjust a learning parameter based on a first performance determined based on the first action and the target action; and

during a subsequent iteration;

causing the apparatus to generate a second control signal based on the sensory context and the adjusted learning parameter;

causing the apparatus to execute a second action based on the second control signal, the second action being characterized by a second performance;

wherein;

the plurality of iterations is configured to occur within a time interval characterized by a duration;

the first iteration and the subsequent iteration are separated by the duration;

a provision of the user input during the subsequent iteration is configured to cause the controller to execute a third action characterized by a third performance value; and

the second action is configured closer to the target action as compared to the third action.

21. The method of claim 20, wherein;

the performance measure being determined by the apparatus; and

the first performance value is lower than the second performance value.

22. The method of claim 20, wherein;

the learning parameter adjustment is configured based on a supervised learning process configured based on the sensory context and a combination of the control signal and the user input; and

the third performance value is lower than the second performance value.