Interactive complex task teaching system that allows for natural language input, recognizes a user's intent, and automatically performs tasks in document object model (DOM) nodes
First Claim
1. An interactive method for learning and executing executable tasks using language and demonstration inputs from a user, comprising:
- a. providing a computational device including a graphical user interface (GUI);
b. providing software running on said computational device and supported GUI-based interaction with said user;
c. wherein said user performs tasks using the GUId. wherein said software includes a natural dialog-based interface whereby said user can communicate with said software using natural dialog-based language;
e. for each of said executable tasks, recognizing said user'"'"'s overall intent;
f. for each of said executable tasks, identifying a plurality of steps needed to complete said task;
g. for each of said plurality of steps, identifying and generalizing a step objective;
h. learning to execute each of said steps from demonstrations provided by said user;
i. providing incremental execution and interaction with said user using said natural dialog-based interface;
j. providing a database for storing a task definition for each of said tasks, wherein said task definition includes said steps comprising said task and said step objectives;
k. storing said task definitions in said database;
l. for each of said tasks, learning semantic characterization of said task for later retrieval from said database;
m. retrieving a particular task definition from said database using said semantic characterization;
n. improving said task definition for each of said tasks through practice, with instruction being provided by said user;
o. wherein one of said executable tasks returns a list of results;
p. displaying said list of results in a first configuration of said GUI wherein said GUI displays a list consisting of multiple Document Object Model nodes;
q. wherein said user provides natural language to said software running on said computational device indicating that iteration should be performed;
r. automatically creating a second configuration of said GUI wherein said GUI displays a plurality of cells arranged into columns and rows, with each row representing a single Document Object Model node from said displayed list;
s. wherein said user demonstrates a first task to be performed in a first Document Object Model node; and
t. wherein thereafter said software running on said computation device automatically performs said first task demonstrate by said user in all other Document Object Model nodes and displays a result of said performance in said second configuration of said GUI.
1 Assignment
0 Petitions
Accused Products
Abstract
A system which allows a user to teach a computational device how to perform complex, repetitive tasks that the user usually would perform using the device'"'"'s graphical user interface (GUI) often but not limited to being a web browser. The system includes software running on a user'"'"'s computational device. The user “teaches” task steps by inputting natural language and demonstrating actions with the GUI. The system uses a semantic ontology and natural language processing to create an explicit representation of the task that is stored on the computer. After a complete task has been taught, the system is able to automatically execute the task in new situations. Because the task is represented in terms of the ontology and user'"'"'s intentions, the system is able to adapt to changes in the computer code while still pursuing the objectives taught by the user.
391 Citations
17 Claims
-
1. An interactive method for learning and executing executable tasks using language and demonstration inputs from a user, comprising:
-
a. providing a computational device including a graphical user interface (GUI); b. providing software running on said computational device and supported GUI-based interaction with said user; c. wherein said user performs tasks using the GUI d. wherein said software includes a natural dialog-based interface whereby said user can communicate with said software using natural dialog-based language; e. for each of said executable tasks, recognizing said user'"'"'s overall intent; f. for each of said executable tasks, identifying a plurality of steps needed to complete said task; g. for each of said plurality of steps, identifying and generalizing a step objective; h. learning to execute each of said steps from demonstrations provided by said user; i. providing incremental execution and interaction with said user using said natural dialog-based interface; j. providing a database for storing a task definition for each of said tasks, wherein said task definition includes said steps comprising said task and said step objectives; k. storing said task definitions in said database; l. for each of said tasks, learning semantic characterization of said task for later retrieval from said database; m. retrieving a particular task definition from said database using said semantic characterization; n. improving said task definition for each of said tasks through practice, with instruction being provided by said user; o. wherein one of said executable tasks returns a list of results; p. displaying said list of results in a first configuration of said GUI wherein said GUI displays a list consisting of multiple Document Object Model nodes; q. wherein said user provides natural language to said software running on said computational device indicating that iteration should be performed; r. automatically creating a second configuration of said GUI wherein said GUI displays a plurality of cells arranged into columns and rows, with each row representing a single Document Object Model node from said displayed list; s. wherein said user demonstrates a first task to be performed in a first Document Object Model node; and t. wherein thereafter said software running on said computation device automatically performs said first task demonstrate by said user in all other Document Object Model nodes and displays a result of said performance in said second configuration of said GUI. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)
-
Specification