System and method for reinforcing programming education through robotic feedback

• US 9,370,862 B2
• Filed: 06/11/2015
• Issued: 06/21/2016
• Est. Priority Date: 06/12/2014
• Status: Active Grant

• Current Assignee: Wonder Workshop, Inc.
• Original Assignee: Play-I, Inc.
• Inventors: Gupta, Saurabh, Gupta, Vikas
• Primary Examiner(s): Amin, Bhavesh V
• Application Number:
  US14/737,342
• Publication Number:
  US 20150364060A1
• Time in Patent Office: 376 Days
• US Class Current: 1/1
• CPC Class Codes: A63H 11/00 Self-movable toy figures
• CPC Class Codes: A63H 2200/00 Computerized interactive toys, e.g. dolls
• CPC Class Codes: A63H 3/28 Arrangements of sound-producing means in dolls; Means in dolls for producing sounds
• CPC Class Codes: A63H 30/04 using wireless transmission
• CPC Class Codes: B25J 11/001 with emotions simulating means
• CPC Class Codes: B25J 13/06 Control stands, e.g. consoles, switchboards
• CPC Class Codes: B25J 9/0081 with master teach-in means
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G09B 19/0053 Computers, e.g. programming
• CPC Class Codes: Y10S 901/04 Manual lead through

Abstract

A method for reinforcing programming education through toy robot feedback, including: at a user device, remote from the toy robot: receiving a series of programming inputs from a user at a programming interface application on the user device; receiving a set of sensor measurements from the toy robot; automatically generating a set of control instructions for the toy robot based on a programming input of the series and the set of sensor measurements; and sending the set of control instructions to the toy robot; and, at the toy robot: receiving the set of control instructions from the user device; operating the toy robot based on the set of control instructions; recording a second set of sensor measurements during toy robot operation; and sending the second set of sensor measurements to the user device.

Claims

1. A method for toy robot remote control using a user device, the toy robot including a set of sensors and a processor configured to control toy robot operation, the method comprising, at the user device:

a) receiving a series of programming inputs from a user at the user device;

b) receiving a set of sensor measurements from the remote toy robot;

c) automatically generating a set of control instructions based on the set of sensor measurements and an unperformed programming input of the series;

d) sending the set of control instructions to the toy robot, wherein the toy robot is operated based on the set of control instructions;

e) iteratively repeating b) to d) for successive programming inputs based on subsequent sets of sensor measurements received from the toy robot, wherein the subsequent sets of sensor measurements are recorded during to robot operation based on a previous set of control instructions and monitoring toy robot performance of the previous set of control instructions based on the subsequent set of sensor measurements received from the toy robot, comprising;

iteratively repeating b) to d) for the successive programming input in response to the toy robot performing the previous set of control instructions within a threshold degree of error;

identifying an inconsistency between the subsequent set of sensor measurements and a requirement for a successive set of control instructions; and

in response to identification of an inconsistency;

determining a secondary programming input based on the subsequent set of sensor measurements;

displaying a virtual representation of the secondary programming input;

receiving a selection of the secondary programming input;

replacing the successive programming component with the secondary programming input; and

performing b) to d) using the secondary programming input as the unperformed programming component of the series.

2. The method of claim 1, wherein the programming inputs are received at a programming interface application on the user device, wherein the programming interface application comprises a visual programming interface.

3. The method of claim 2, wherein the programming inputs comprise programming components.

4. The method of claim 2, wherein the programming inputs comprise virtual, connectable blocks, each virtual block representing at least one of a robot movement, robot audio output, and robot visual output.

5. The method of claim 2, wherein the programming interface comprises an animation programming interface, wherein the programming interface automatically converts a virtual driving path drawn by the user within the programming interface and conditional events associated with the virtual driving path into the set of programming inputs.

6. The method of claim 2, wherein the set of control instructions is automatically generated in response to receipt of a new programming input from the user, wherein the set of sensor measurements is received after the new programming input is received, and the set of control instructions is generated based on the new programming input and the set of sensor measurements.

7. The method of claim 1, further comprising, at the user device:

determining performance of the previous set of control instructions by the toy robot based on the set of sensor measurements received from the toy robot after sending the set of control instructions to the toy robot.

9. The method of claim 1, wherein the subsequent set of sensor measurements are received by the user device as a near-real time sensor measurement stream from the toy robot.

8. A method for reinforcing programming education through toy robot feedback, the toy robot including a set of sensors, the method comprising:

at a user device, remote from the toy robot;

a) receiving a series of programming inputs from a user at a programming interface application on the user device;

b) receiving a set of sensor measurements from the toy robot;

c) automatically generating a set of control instructions for the toy robot based on a programming input of the series and the set of sensor measurements;

d) sending the set of control instructions to the toy robot;

at the toy robot;

e) receiving the set of control instructions from the user device;

f) operating the toy robot based on the set of control instructions, wherein the toy robot is operated based on the first set of control instructions within a threshold time period after first set of control instruction receipt at the toy robot;

g) recording a second set of sensor measurements during toy robot operation;

h) sending the second set of sensor measurements to the user device;

at the user device;

determining execution of the set of control instructions by the toy robot based on data received from the toy robot;

automatically generating a second set of control instructions for the toy robot based on a succeeding programming input of the series and the second set of sensor measurements;

sending the second set of control instructions to the toy robot; and

operating the toy robot according to the second set of control instructions, wherein the toy robot is operated based on the second set of control instructions within the threshold time period after second set of control instruction receipt at the toy robot.

10. The method of claim 8, wherein determining execution of the programming input by the toy robot based on data received from the toy robot comprises determining execution of the programming input based on the second set of sensor measurements.

11. The method of claim 8, wherein the set of control instructions is automatically generated in response to receipt of a user input at the toy robot.

12. The method of claim 8, wherein sending the second set of sensor measurements comprises streaming the sensor measurements from the toy robot to the user device in near-real time.

13. The method of claim 12, further comprising:

at the user device;

determining an inconsistency between the second set of sensor measurements and a requirement for the succeeding programming input;

sending the toy robot a pause control instruction;

displaying a first and second programming input option on the user device;

receiving a programming input option selection of the first or second programming input option;

generating and sending new control instructions based on the selected programming input option to the toy robot; and

at the toy robot;

receiving the new control instructions; and

operating the toy robot based on the new control instructions.

14. The method of claim 8, wherein the programming inputs comprise conditional toy robot actions, the method further comprising:

sending the programming inputs to the toy robot;

storing the programming inputs at the toy robot as a robot personality; and

at the toy robot, automatically operating the toy robot based on the robot personality.

15. The method of claim 14, wherein a) to h) is performed in response to the toy robot wirelessly connecting to the user device, wherein the toy robot is automatically operated based on the robot personality independent of user device connection.

16. The method of claim 14, further comprising:

receiving a second series of programming inputs at a user device;

receiving a third set of sensor measurements from the toy robot at the user device;

generating a third set of control instructions based on the sensor measurements at the user device;

sending the third set of control instructions to the toy robot from the user device;

receiving the third set of control instructions at the toy robot;

modifying the third set of control instructions into modified control instructions based on the robot personality; and

operating the toy robot based on the modified control instructions.

17. The method of claim 8, wherein the programming interface comprises an animation programming interface, wherein the programming interface automatically converts a virtual driving path drawn by the user within the programming interface and conditional events associated with the virtual driving path into the set of programming inputs, wherein the animation programming interface comprises a virtual grid representative of a physical environment adjacent the toy robot and a to-scale virtual representation of the toy robot.

18. A method for toy robot operation, using a user device, to reinforce programming education through toy robot feedback, the toy robot including a set of sensors, the method comprising, at the toy robot:

automatically operating the toy robot in a standby mode, comprising;

recording a set of sensor measurements;

operating based on stored control instructions and the set of sensor measurements;

in response to wireless connection with a user device;

recording a second set of sensor measurements;

transmitting the second set of sensor measurements to the user device;

receiving a set of control instructions from the user device, the set of control instructions automatically generated by the user device based on the second set of sensor measurements and a programming component stored by the user device;

modifying a control instruction of the received set of instructions based on the stored control instructions;

automatically executing the modified set of control instructions;

recording a third set of sensor measurements during control instruction execution;

at the user device;

storing a history of programming components associated with the toy robot;

identifying a pattern of conditional control instructions from the history of programming components;

generating updated control instructions for the toy robot based on the identified pattern; and

sending the updated control instructions to the robot;

at the toy robot;

receiving and storing the updated control instructions; and

operating based on the updated control instructions.

19. The method of claim 18, further comprising, at the toy robot:

recording a third set of sensor measurements;

transmitting the third set of sensor measurements to the user device;

receiving a third set of control instructions from the user device, the third set of control instructions automatically generated by the user device based on the second set of sensor measurements and a programming component stored by the user device; and

automatically executing the third set of control instructions.

20. The method of claim 19, further comprising, at the user device:

receiving a series of programming inputs, wherein the third set of control instructions are generated based on an unperformed programming input of the series;

verifying execution of the third set of control instructions by the toy robot based on a fourth set of sensor measurements received from the toy robot;

automatically generating a fourth set of control instructions for the toy robot based on a succeeding programming input of the series and the fourth set of sensor measurements; and

sending the fourth set of control instructions to the toy robot for execution.