Interactive complex task teaching system

• US 20090119587A1
• Filed: 11/02/2007
• Published: 05/07/2009
• Est. Priority Date: 11/02/2007
• Status: Active Grant

• Current Assignee: Florida Institute For Human And Machine Cognition, Inc.
• Original Assignee: Florida Institute For Human And Machine Cognition, Inc.
• Inventors: Jung, Hyuckchul, Allen, James F., Galescu, Lucian, Taysom, William, Chambers, Nathanael
• Granted Patent: US 7,983,997 B2
• Time in Patent Office: Days
• US Class Current: 715/705
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A system which allows a user to teach a computational device how to perform complex, repetitive tasks that the user usually would perform using the device'"'"'s graphical user interface (GUI) often but not limited to being a web browser. The system includes software running on a user'"'"'s computational device. The user “teaches” task steps by inputting natural language and demonstrating actions with the GUI. The system uses a semantic ontology and natural language processing to create an explicit representation of the task that is stored on the computer. After a complete task has been taught, the system is able to automatically execute the task in new situations. Because the task is represented in terms of the ontology and user'"'"'s intentions, the system is able to adapt to changes in the computer code while still pursuing the objectives taught by the user.

Claims

1. An interactive method for learning and executing executable tasks using language and demonstration inputs from a user, comprising:

a. providing a computational device including a graphical user interface (GUI);

b. providing software running on said computational device and supported GUI-based interaction with said user;

c. wherein said user performs tasks using the GUI

d. wherein said software includes a natural dialog-based interface whereby said user can communicate with said software using natural dialog-based language;

e. for each of said executable tasks, recognizing said user'"'"'s overall intent;

f. for each of said executable tasks, identifying a plurality of steps needed to complete said task;

g. for each of said plurality of steps, identifying and generalizing a step objective;

h. learning to execute each of said steps from demonstrations provided by said user;

i. providing incremental execution and interaction with said user using said natural dialog-based interface;

j. providing a database for storing a task definition for each of said tasks, wherein said task definition includes said steps comprising said task and said step objectives;

k. storing said task definitions in said database;

l. for each of said tasks, learning semantic characterization of said task for later retrieval from said database;

m. retrieving a particular task definition from said database using said semantic characterization; and

n. improving said task definition for each of said tasks through practice, with instruction being provided by said user.

2. An interactive method for learning and executing as recited in claim 1, wherein the step of recognizing said user'"'"'s overall intent for each of said executable tasks comprises:

a. using an ontology of tasks;

b. characterizing said user'"'"'s actions within said ontology by observing said user'"'"'s actions, objects with which said user interacts, as well as natural dialog-based language descriptions of said actions and said objects provided by said user;

c. using algorithms based on natural language grammars and lexicons to map sentences in said dialog to said ontology; and

d. providing an ontology-driven learning of executable steps from observations of said user'"'"'s actions.

3. An interactive method for learning and executing as recited in claim 1, wherein the step of providing software which includes a natural dialog-based interface whereby said user can communicate with said software using natural dialog-based language comprises:

a. describing said tasks and said steps comprising said tasks in natural language;

b. providing a grammar and lexicon with mapping rules to said ontology;

c. using natural language processing techniques to convert said natural language provided by said user into a representation based on the ontology;

d. having said software pose clarifying questions to said user expressed in natural language when needed;

e. summarizing said task steps in natural language as said task steps are learned; and

f. adding to said lexicon as new unknown words appear in said natural dialog-based interface.

4. An interactive method for learning and executing as recited in claim 1, further comprising:

a. detecting the start of each new step from language communicated by said user and observed user actions in said GUI;

b. for each new step identifying the type of action required in said step in said ontology;

c. for each new step, using the language describing said step to identify one or more parameters to be used in said step;

d. for each new step, if the intention of said new step is unclear, querying said user for additional information; and

e. once a step is clearly defined, adding said step to said database.

5. An interactive method for learning and executing as recited in claim 2, further comprising:

a. observing actions taken by said user in said GUI;

b. retrieving internal encodings of GUI elements used in said observed user actions;

c. defining a correlation between said descriptions provided by said user and said values entered or selected by said user in said actions;

d. verifying learned patterns and interacting with said user when problems arise.

6. An interactive method for learning and executing as recited in claim 5, wherein said step of verifying learned patterns and interacting with said user when problems arise comprises:

a. simulating execution of said learned steps;

b. in the event a problem arises, notifying said user of said problem; and

c. accepting additional examples or descriptions from said user in order to correct said problem.

7. An interactive method for learning and executing as recited in claim 1, further comprising:

a. detecting the presence of iteration;

b. automatically identifying the individual elements in lists or tables;

c. presenting said lists or tables in tabular form;

d. receiving instruction from said user, wherein said user demonstrates an appropriate action in one element in said table;

e. automatically replicating said user'"'"'s demonstrated appropriate action as to all other elements in said table;

f. receiving additional instruction from said user, wherein said user demonstrates an appropriate action as to other elements in said table in order to correct any errors made; and

g. monitoring said user'"'"'s language in order to detect an explicit statement of termination, whereupon the process of iteration is terminated.

8. An interactive method for learning and executing as recited in claim 7, wherein said step of detecting the presence of iteration comprises analyzing the explicit language used by said user.

9. An interactive method for learning and executing as recited in claim 2, further comprising:

a. upon receiving an indication from said user that a new task is to be learned, using said user'"'"'s linguistic description to classify said new task into one of said task-specific ontologies;

b. identifying input and output parameters for said new task according to the way said parameters were described by said user;

c. allowing said user to explicitly describe additional parameters; and

d. querying said user for clarification when said user'"'"'s intent is not identified.

10. An interactive method for learning and executing as recited in claim 1, further comprising:

a. receiving from said user a description of a task to be performed;

b. encoding said described task in terms of task ontology;

c. using said encoding to search said database in order to identify said task within said database;

d. retrieving said task from said database, along with parameters needed to perform said task; and

e. querying said user for values for any of said parameters which need to be specified.

11. An interactive method for learning and executing as recited in claim 7, wherein said step of detecting the presence of iteration comprises identifying lists and tables.

12. An interactive method for learning and executing as recited in claim 1, wherein for each of said plurality of steps taught by said user, said demonstrations of said user and a primary code object which is associated with said demonstrations of said user are correlated.

14. An interactive method for learning and executing as recited in claim 12, further comprising for each of said plurality of steps taught by said user, scanning other code objects in proximity to said primary code object which is associated with a particular said user demonstration to search for the presence of words defining said user'"'"'s objective or synonyms therefor.

15. An interactive method for learning and executing as recited in claim 14, further comprising:

a. for each of said plurality of steps taught by said user, creating a primary link between a particular said user demonstration and said primary code object and also creating a secondary link between said particular user demonstration and said other code objects which relate to said word'"'"'s defining said user'"'"'s objective or synonyms therefor; and

b. saving said primary and secondary links in said database.

16. An interactive method for learning and executing as recited in claim 2, wherein for each of said plurality of steps taught by said user, said demonstrations of said user and a primary code object which is associated with said demonstrations of said user are correlated.

17. An interactive method for learning and executing as recited in claim 16, further comprising for each of said plurality of steps taught by said user, scanning other code objects in proximity to said primary code object which is associated with a particular said user demonstration to search for the presence of words defining said user'"'"'s objective or synonyms therefor.

18. An interactive method for learning and executing as recited in claim 17, further comprising:

a. for each of said plurality of steps taught by said user, creating a primary link between a particular said user demonstration and said primary code object and also creating a secondary link between said particular user demonstration and said other code objects which relate to said word'"'"'s defining said user'"'"'s objective or synonyms therefor; and

b. saving said primary and secondary links in said database.

19. An interactive method for learning and executing as recited in claim 3, wherein for each of said plurality of steps taught by said user, said demonstrations of said user and a primary code object which is associated with said demonstrations of said user are correlated.

20. An interactive method for learning and executing as recited in claim 19, further comprising for each of said plurality of steps taught by said user, scanning other code objects in proximity to said primary code object which is associated with a particular said user demonstration to search for the presence of words defining said user'"'"'s objective or synonyms therefor.