Method for aligning demonstrated user actions with existing documentation

• US 8,607,142 B2
• Filed: 03/03/2008
• Issued: 12/10/2013
• Est. Priority Date: 11/19/2004
• Status: Active Grant

• Current Assignee: Activision Publishing Incorporated (Microsoft Corporation)
• Original Assignee: Activision Publishing Incorporated (Microsoft Corporation)
• Inventors: Bergman, Lawrence D., Castelli, Vittorio, Creswick, Eugene Rogan, Lau, Tessa A., Oblinger, Daniel A.
• Primary Examiner(s): Chow, Dennis-Doon
• Assistant Examiner(s): CHOI, DAVID E
• Application Number:
  US12/041,142
• Publication Number:
  US 20080155393A1
• Time in Patent Office: 2,108 Days
• Field of Search: 715/710, 715/709, 715/708, 715/705
• US Class Current: 715/705
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40

Abstract

There is provided an apparatus for aligning document actions specified in a documented procedure with demonstrated user actions. The apparatus includes an alignment device for computing an alignment between the document actions and the demonstrated user actions based on at least one similarity metric.

Claims

1. A method of aligning document actions specified in a documented procedure with demonstrated user actions, comprising:

determining, by one or more processors, a plurality of document actions specified in the documented procedure that are to be taken by a user and a plurality of demonstrated user actions performed in a user interface;

computing by the one or more processors, an alignment between the plurality of document actions and the plurality of demonstrated user actions based on at least one similarity metric;

determining, by the one or more processors, a document action of the plurality of document actions that corresponds to a demonstrated user action being performed in the user interface; and

determining, by the one or more processors, a next document action from the plurality of document actions that is to be taken based on the alignment.

2. The method of claim 1, wherein the document action is associated with a current step in the documented procedure that is to be taken by a user, and wherein the next document action is associated with a next step in the documented procedure that is to be taken by the user.

3. The method of claim 1, wherein the documented procedure is displayed on the user interface, the method further comprising:

highlighting, by the one or more processors, the document action on the user interface as a current step being taken.

4. The method of claim 1, wherein the documented procedure is displayed on the user interface, the method further comprising:

highlighting, by the one or more processors, the next document action in the user interface as a next step to be taken.

5. The method of claim 1, wherein the documented procedure is displayed on the user interface, the method further comprising:

highlighting, by the one or more processors, the document action in the user interface as a current step being taken; and

highlighting, by the one or more processors, the next document action in the user interface as a next step to be taken,

wherein the document action is highlighted in a first manner, and wherein the next document action is highlighted in a second manner that is different than the first manner.

6. The method of claim 1, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action.

7. The method of claim 1, wherein the at least one similarity metric is based on a probability distribution over at least two steps of the documented procedure.

8. The method of claim 1, wherein the at least one similarity metric is based on a similarity between an action type of the demonstrated user action and an action type of the document action.

9. The method of claim 1, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action, and a probability distribution over at least two steps of the documented procedure.

10. The method of claim 1, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action, and a similarity between an action type of the demonstrated user action and an action type of the document action.

11. The method of claim 1, wherein the at least one similarity metric is based on a probability distribution over at least two steps of the documented procedure, and a similarity between an action type of the demonstrated user action and an action type of the document action.

12. The method of claim 1, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action, a probability distribution over at least two steps of the documented procedure, and a similarity between an action type of the demonstrated user action and an action type of the document action.

13. The method of claim 1, wherein the document action is determined to correspond to the demonstrated user action based on the alignment.

14. The method of claim 1, further comprising:

capturing, by a camera communicatively coupled to the one or more processors, one or more images of the demonstrated user action during performance of the demonstrated user action,

wherein the document action is determined to correspond to the demonstrated user action based on the one or more images.

15. The method of claim 1, further comprising:

capturing, by the camera, images of the plurality of demonstrated user actions during performance of the plurality of demonstrated user actions,

wherein the alignment is computed further based on the images.

16. A system for aligning document actions specified in a documented procedure with demonstrated user actions, comprising:

one or more processors configured to;

determine a plurality of document actions specified in the documented procedure that are to be taken by a user and a plurality of demonstrated user actions performed in a user interface;

compute an alignment between the plurality of document actions and the plurality of demonstrated user actions based on at least one similarity metric;

determine a document action of the plurality of document actions that corresponds to a demonstrated user action being performed in the user interface; and

determine a next document action from the plurality of document actions that is to be taken based on the alignment.

17. The system of claim 16, wherein the document action is associated with a current step in the documented procedure that is to be taken by a user, and wherein the next document action is associated with a next step in the documented procedure that is to be taken by the user.

18. The system of claim 16, wherein the documented procedure is displayed on the user interface, and wherein the one or more processors are configured to:

highlight the document action in the user interface as a current step being taken.

19. The system of claim 16, wherein the documented procedure is displayed on the user interface, and wherein the one or more processors are configured to:

highlight the next document action in the user interface as a next step to be taken.

20. The system of claim 16, wherein the documented procedure is displayed on the user interface, and wherein the one or more processors are configured to:

highlight the document action in the user interface as a current step being taken; and

highlight the next document action in the user interface as a next step to be taken,

wherein the document action is highlighted in a first manner, and wherein the next document action is highlighted in a second manner that is different than the first manner.

21. The system of claim 16, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action.

22. The system of claim 16, wherein the at least one similarity metric is based on a probability distribution over at least two steps of the documented procedure.

23. The system of claim 16, wherein the at least one similarity metric is based on a similarity between an action type of the demonstrated user action and an action type of the document action.

24. The system of claim 16, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action, and a probability distribution over at least two steps of the documented procedure.

25. The system of claim 16, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action, and a similarity between an action type of the demonstrated user action and an action type of the document action.

26. The system of claim 16, wherein the at least one similarity metric is based on a probability distribution over at least two steps of the documented procedure, and a similarity between an action type of the demonstrated user action and an action type of the document action.

27. The system of claim 16, wherein the at least one similarity metric is based on one or more words in common between a textual description of the demonstrated user action and the document action, a probability distribution over at least two steps of the documented procedure, and a similarity between an action type of the demonstrated user action and an action type of the document action.

28. The system of claim 16, wherein the document action is determined to correspond to the demonstrated user action based on the alignment.

29. The system of claim 16, further comprising a camera communicatively coupled to the one or more processors, wherein the camera is configured to capture one or more images of the demonstrated user action during performance of the demonstrated user action, and wherein the document action is determined to correspond to the demonstrated user action based on the one or more images.

30. The system of claim 16, wherein the camera is configured to capture images of the plurality of demonstrated user actions during performance of the plurality of demonstrated user actions, and wherein the alignment is computed further based on the images.

31. A non-transitory computer readable storage media comprising executable code for aligning document actions specified in a documented procedure with demonstrated user actions, the computer executable code, when executed by one or more processors, causing the one or more processors to:

determine a plurality of document actions specified in the documented procedure that are to be taken by a user and a plurality of demonstrated user actions performed in a user interface;

compute an alignment between the plurality of document actions and the plurality of demonstrated user actions based on at least one similarity metric;

determine a document action of the plurality of document actions that corresponds to a demonstrated user action being performed in the user interface; and

determine a next document action from the plurality of document actions that is to be taken based on the alignment.