Method and apparatus for automated assistance with task management

• US 8,694,355 B2
• Filed: 06/01/2009
• Issued: 04/08/2014
• Est. Priority Date: 05/30/2008
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Bui, Hung, Eker, Steven, Elenius, Daniel, Gervasio, Melinda, Lee, Thomas J., Marker, Mei, Morley, David, Murdock, Janet, Myers, Karen, Peintner, Bart, Saadati, Shahin, Yeh, Eric, Yorke-Smith, Neil
• Primary Examiner(s): LOFTIS, JOHNNA RONEE
• Application Number:
  US12/476,020
• Publication Number:
  US 20090307162A1
• Time in Patent Office: 1,772 Days
• Field of Search: 705/7.27
• US Class Current: 705/7.27
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

The present invention relates to a method and apparatus for assisting with automated task management. In one embodiment, an apparatus for assisting a user in the execution of a task, where the task includes one or more workflows required to accomplish a goal defined by the user, includes a task learner for creating new workflows from user demonstrations, a workflow tracker for identifying and tracking the progress of a current workflow executing on a machine used by the user, a task assistance processor coupled to the workflow tracker, for generating a suggestion based on the progress of the current workflow, and a task executor coupled to the task assistance processor, for manipulating an application on the machine used by the user to carry out the suggestion.

Claims

1. A method for assisting a user in an execution of a task, where the task comprises one or more workflows required to accomplish a goal defined by the user, the method comprising:

identifying, using a workflow tracker, a current workflow executing on a machine used by the user, where the current workflow is one of the one or more workflows and wherein the identifying is performed automatically by the machine in an absence of a manual identification of the current workflow by matching data related to the current workflow to a stored task model, wherein the identifying comprises;

receiving, from one or more instrumented applications executing on the machine used by the user, the data related to the current workflow; and

identifying the stored task model in a workflow library as one of a plurality of task models that most closely matches the current workflow, based on the data, wherein the stored task model encodes a set of workflows for accomplishing a corresponding task;

tracking, using the workflow tracker, a progress of the current workflow in accordance with the stored task model, wherein the stored task model comprises a variablized version of a previous workflow that is customized during the execution for the current workflow;

generating a suggestion based on the progress of the current workflow and on the stored task model, where the suggestion assists the user in progressing toward a completion of the task;

manipulating an application on the machine used by the user to carry out the suggestion; and

learning the stored task model from a user demonstration, wherein the learning comprises;

recording an action sequence that embodies a dataflow, in accordance with the user demonstration, wherein the action sequence comprises one or more observed actions executed on the machine used by the user; and

inserting at least one inferred action into the dataflow, wherein the at least one inferred action is inserted in a gap between a first one of the one or more observed actions and a second one of the one or more observed actions, and wherein the at least one inferred action is not one of the one or more observed actions,

wherein at least one of;

the workflow tracking, the generating, or the manipulating is performed using a processor.

2. The method of claim 1, wherein the at least one inferred action produces an output to support an input required by the second one of the one or more observed actions, and wherein the input is not supported by a previous one of the one or more observed actions.

3. The method of claim 1, wherein the inserting comprises:

querying a knowledge base for one or more potential paths that bridge the gap, wherein each of the one or more potential paths comprises one or more actions;

presenting the one or more potential paths to the user;

receiving a selection of one of the one or more potential paths from the user, wherein the one of the one or more potential paths comprises the at least one inferred action; and

completing the dataflow in accordance with the one of the one or more potential paths.

4. The method of claim 3, further comprising:

filtering the one or more potential paths in accordance with one or more criteria, prior to presenting the one or more potential paths to the user.

5. The method of claim 4, wherein the one or more criteria comprises at least one of:

a number of inputs or outputs to which each of the one or more potential paths is connected, a redundancy of the one or more potential paths, a repetition of sub-paths within the one or more potential paths, an occurrence of subsumable paths within the one or more potential paths, or commonalties among the one or more potential paths.

6. The method of claim 3, further comprising:

presenting the dataflow, as completed, to the user;

receiving user feedback regarding the dataflow, as completed;

adjusting the dataflow based on the user feedback; and

storing the dataflow as a task model in the workflow library.

7. The method of claim 6, wherein the user feedback comprises an answer to a question.

8. The method of claim 1, wherein the tracking comprises:

identifying a step of the stored task model that corresponds to a current step of the current workflow; and

following the progress of the current workflow in accordance with the set of workflows encoded in the stored task model, starting from the step of the stored task model.

9. The method of claim 8, wherein the generating comprises:

determining when the current workflow satisfies a trigger condition of an assistance pattern;

identifying an action corresponding to the trigger condition; and

forming the suggestion, wherein the suggestion suggests executing the action.

10. The method of claim 9, wherein the manipulating comprises:

executing the action.

11. The method of claim 10, wherein the manipulating further comprises:

transmitting the suggestion to the user; and

waiting for approval of the suggestion by the user before executing the action.

12. The method of claim 1, wherein the tracking comprises representing the current workflow as a Logical Hidden Markov Model.

13. A method for assisting a user in an execution of a task, where the task comprises one or more workflows required to accomplish a goal defined by the user, the method comprising:

identifying, using a workflow tracker, a current workflow executing on a machine used by the user, where the current workflow is one of the one or more workflows and wherein the identifying is performed automatically by the machine in an absence of a manual identification of the current workflow by matching data related to the current workflow to a stored task model, wherein the identifying comprises;

receiving, from one or more instrumented applications executing on the machine used by the user, the data related to the current workflow; and

identifying the stored task model in a workflow library as one of a plurality of task models that most closely matches the current workflow, based on the data, wherein the stored task model encodes a set of workflows for accomplishing a corresponding task;

tracking, using the workflow tracker, a progress of the current workflow in accordance with the stored task model, wherein the stored task model comprises a variablized version of a previous workflow that is customized during the execution for the current workflow;

generating a suggestion based on the progress of the current workflow and on the stored task model, where the suggestion assists the user in progressing toward a completion of the task;

manipulating an application on the machine used by the user to carry out the suggestion; and

learning the stored task model from a user demonstration, wherein the learning comprises;

discovering a loop in the user demonstration previous workflow, wherein the user demonstration previous workflow comprises a plurality of actions, and wherein the loop comprises at least one repeated sequence of actions occurring in the plurality of actions,

wherein at least one of;

the workflow tracking, the generating, or the manipulating is performed using a processor.

14. The method of claim 13, wherein the discovering comprises, for each action in the plurality of actions:

variablizing the action;

exploring at least one loop anchored at the action; and

deriving a set of one or more generalized program fragments from the at least one loop.

15. The method of claim 14, further comprising:

grafting the set of one or more generalized program fragments onto a current generalized program fragment for all of the plurality of actions up to, but not including, an action corresponding to an index into the user demonstration that is associated with a start of the body of the at least one loop.

16. The method of claim 14, wherein the exploring comprises:

receiving a plurality of inputs, wherein the plurality of inputs comprises;

a length of a body of the at least one loop, an index into the user demonstration that is associated with a start of the body of the at least one loop, a starting point index into the user demonstration, and a multimap that that maps one or more data values to one or more supports for the action;

constructing the at least one loop in accordance with the plurality of inputs;

executing the at least one loop for a plurality of iterations, wherein a number of the plurality of iterations is one less than a current iteration count; and

forming a loop statement in accordance with variables contained in the at least one loop.

17. The method of claim 13, wherein the loop loops over at least one collection of objects.

18. The method of claim 17, wherein the at least one collection of objects comprises one unordered collection of objects or one or more ordered collections of objects.

19. The method of claim 17, wherein the loop outputs an ordered collection of objects.

20. The method of claim 13, wherein the tracking comprises representing the current workflow as a Logical Hidden Markov Model.

21. The method of claim 13, wherein the tracking comprises:

identifying a step of the stored task model that corresponds to a current step of the current workflow; and

following the progress of the current workflow in accordance with the set of workflows encoded in the stored task model, starting from the step of the stored task model.

22. The method of claim 21, wherein the generating comprises:

determining when the current workflow satisfies a trigger condition of an assistance pattern;

identifying an action corresponding to the trigger condition; and

forming the suggestion, wherein the suggestion suggests executing the action.

23. The method of claim 22, wherein the manipulating comprises:

executing the action.

24. The method of claim 23, wherein the manipulating further comprises:

transmitting the suggestion to the user; and

waiting for approval of the suggestion by the user before executing the action.

25. A computer readable storage medium containing an executable program for assisting a user in an execution of a task, where the task comprises one or more workflows required to accomplish a goal defined by the user, where the program performs steps of:

identifying, using a workflow tracker, a current workflow executing on a machine used by the user, where the current workflow is one of the one or more workflows and wherein the identifying is performed automatically by the machine in an absence of a manual identification of the current workflow by matching data related to the current workflow to a stored task model, wherein the identifying comprises;

receiving, from one or more instrumented applications executing on the machine used by the user, the data related to the current workflow; and

identifying the stored task model in a workflow library as one of a plurality of task models that most closely matches the current workflow, based on the data, wherein the stored task model encodes a set of workflows for accomplishing a corresponding task;

tracking, using the workflow tracker, a progress of the current workflow in accordance with the stored task model, wherein the stored task model comprises a variablized version of a previous workflow that is customized during the execution for the current workflow;

generating a suggestion based on the progress of the current workflow and on the task model, where the suggestion assists the user in progressing toward a completion of the task;

manipulating an application on the machine used by the user to carry out the suggestion; and

learning the stored task model from a user demonstration, wherein the learning comprises;

recording an action sequence that embodies a dataflow, in accordance with the user demonstration, wherein the action sequence comprises one or more observed actions executed on the machine used by the user; and

inserting at least one inferred action into the dataflow, wherein the at least one inferred action is inserted in a gap between a first one of the one or more observed actions and a second one of the one or more observed actions, and wherein the at least one inferred action is not one of the one or more observed actions.

26. A computer readable storage medium containing an executable program for assisting a user in an execution of a task, where the task comprises one or more workflows required to accomplish a goal defined by the user, where the program performs steps of:

identifying, using a workflow tracker, a current workflow executing on a machine used by the user, where the current workflow is one of the one or more workflows and wherein the identifying is performed automatically by the machine in an absence of a manual identification of the current workflow by matching data related to the current workflow to a stored task model, wherein the identifying comprises;

receiving, from one or more instrumented applications executing on the machine used by the user, the data related to the current workflow; and

identifying the stored task model in a workflow library as one of a plurality of task models that most closely matches the current workflow, based on the data, wherein the stored task model encodes a set of workflows for accomplishing a corresponding task;

tracking, using the workflow tracker, a progress of the current workflow in accordance with the stored task model, wherein the stored task model comprises a variablized version of a previous workflow that is customized during the execution for the current workflow;

generating a suggestion based on the progress of the current workflow and on the task model, where the suggestion assists the user in progressing toward a completion of the task;

manipulating an application on the machine used by the user to carry out the suggestion; and

learning the stored task model from a user demonstration, wherein the learning comprises;

discovering a loop in the user demonstration previous workflow, wherein the user demonstration previous workflow comprises a plurality of actions, and wherein the loop comprises at least one repeated sequence of actions occurring in the plurality of actions.