Translating text to, merging, and optimizing graphical user interface tasks

• US 9,189,254 B2
• Filed: 10/01/2010
• Issued: 11/17/2015
• Est. Priority Date: 10/02/2009
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Kushman, Nathaniel, Barzilay, Regina, Branavan, Satchuthananthavale R. Kuhan, Katabi, Dina, Rinard, Martin C.
• Primary Examiner(s): Ng, Amy
• Assistant Examiner(s): VU, TOAN H
• Application Number:
  US12/896,404
• Publication Number:
  US 20110246881A1
• Time in Patent Office: 1,873 Days
• Field of Search: 715/762, 715/708, 715/771, 717/100, 717/124, 717/151
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 9/453 Help systems

Abstract

A system and method that enables a plurality of lay users to collaborate on automating computer tasks is disclosed. In one embodiment, the system automatically performs these tasks, rather than just documenting how to perform them. The system allows a database of solutions to be built for every important computer task. A key characteristic of this system is that users contribute to this database by simply performing the task. The system records the graphical user interface (GUI) actions as the user performs the task. It aggregates GUI traces from multiple users into a canonical sequence of GUI actions parameterized by user-environment that will successfully accomplish the task on a variety of different configurations. A classifier is used to predict which steps are likely to be misinterpreted and requests human intervention to properly perform them. This process can be done iteratively until the translation is believed to be correct.

Claims

1. A method of generating a set of steps to perform a computer based task, comprising:

a. having one or more users perform said task using a respective computing device;

b. delivering one or more traces from said users performing said task to a processing unit, wherein each of said traces represent GUI actions performed on a user interface of said respective computing device, wherein said user interface comprises a series of views, and each of said views comprises a plurality of widgets, each widget configured to accept user input, and wherein said traces comprise a list of GUI actions performed on said plurality of widgets in said series of views; and

c. using said processing unit to execute a set of rules on said traces to generate said set of steps, wherein said set of rules comprises;

mapping each GUI action to an abstract action based on the GUI action'"'"'s effect on the state of the system, by;

determining whether the GUI action causes a change or a pending change to the state of the system, and if it is determined that the GUI action causes a change or pending change to the state of the system, categorizing the GUI action as a state action; and

determining whether the GUI action causes a change in the plurality of visible widgets, and if it is determined that the GUI action causes a change in the plurality of visible widgets, categorizing the GUI action as a navigation action;

optimizing said abstract actions wherein optimizing said abstract actions comprises comparing the abstract actions from said one or more traces and differentiating between differences due to user error and those due to configuration differences based on subsequent views; and

generating said set of steps from said optimized abstract actions.

2. The method of claim 1, wherein said list of GUI actions is obtained using an application resident on said user'"'"'s computing device.

3. The method of claim 2, wherein said application comprises Microsoft Active Accessibility.

4. The method of claim 1, wherein said state of the system comprises the stored value of each of said plurality of widgets.

5. The method of claim 1, wherein determining whether each GUI action causes a change or a pending change to the state of the system comprises using statistical machine learning.

6. The method of claim 5, wherein said machine learning technique comprises a machine learning classifier.

7. The method of claim 1, wherein navigation actions from a trace for one task are used during playback of another task.

8. The method of claim 1, wherein one of said widgets comprises a dialog box, and wherein said first subset of rules specifies that closing a dialog box modifies the state of the system with the values of other widgets in said dialog box.

9. The method of claim 1, wherein optimizing said abstract actions comprises the removal of abstract actions which do not affect the system state.

10. The method of claim 1, wherein optimizing said abstract actions comprises removing mistakes and unnecessary actions from said trace.

11. The method of claim 1, wherein said processing unit receives a plurality of traces from a plurality of users and generates a plurality of sets of steps, further comprising merging said plurality of sets of steps into a single set of steps, parameterized based on configuration differences.

12. The method of claim 11, wherein said set of steps includes modifying the value of a widget, and said rules categorize said value as generic or user specific.

13. The method of claim 1, further comprising executing said generated set of steps on a user'"'"'s computer.

14. The method of claim 1, further comprising:

creating a navigation graph comprising a map of various widgets and paths between said various widgets;

using said navigation graph to create a new set of navigation actions, based on a user'"'"'s configuration;

combining said new set of navigation actions and said optimized abstract actions to form said set of steps; and

executing said set of steps on said user'"'"'s computer.

15. The method of claim 14, wherein said navigation graph is constructed by said processing unit based on said navigation actions found in each of said traces.

16. The method of claim 11, wherein said configuration differences are determined based on differences in said navigation actions found in each of said plurality of traces.

17. The method of claim 1, wherein mapping each GUI action to an abstract action comprises mapping each GUI action to a plurality of abstract actions.

18. The method of claim 1, wherein categorizing the GUI action as a state action further comprises categorizing the GUI action as either an update action or a commit action.

19. The method of claim 18, further comprising using a machine learning classifier to determine whether the state action is an update action or a commit action.

20. The method of claim 18, wherein the machine learning classifier comprises a Support Vector Machine (SVM) classifier.

21. The method of claim 18, further comprising using a first machine learning classifier to determine whether the state action is an update action, and a second machine learning classifier to determine whether the state action is a commit action.

22. The method of claim 18, wherein optimizing said abstract actions comprises removing redundant update actions, filtering unnecessary commit actions, and removing unnecessary navigation actions.