Smart UI recording and playback framework

• US 20060005132A1
• Filed: 06/30/2004
• Published: 01/05/2006
• Est. Priority Date: 06/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Herdeg, Howard B. III
• Granted Patent: US 7,653,896 B2
• Time in Patent Office: Days
• US Class Current: 715/704
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 9/45512 Command shells

Abstract

Techniques and tools are described for recording and reliably replicating graphical user interface (“GUI”) activity for various applications, including applications that have dynamic user interfaces. The techniques and tools may be used in combination or separately. For example, a recording tool receives internal macro data from a GUI-based application as opaque tokens, which are embedded into playback code. A playback tool executes the playback code by passing the embedded tokens back to their original application for playback by the application'"'"'s internal recorder.

Claims

1. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

receiving a plurality of internal macro actions passed as one or more opaque recorded step tokens from a graphical user interface-based application; and

recording the one or more opaque recorded step tokens with an external UI recorder.

2. The computer-readable medium of claim 1 wherein the plurality of internal macro actions are generated by an internal macro recorder of the application and wherein the one or more opaque recorded step tokens are for handling by the external UI recorder to interoperate with the internal macro recorder of the application.

3. The computer-readable medium of claim 1 further comprising:

receiving a plurality of system user interface actions for user interactions against a second graphical user interface-based application;

generating one or more system user interface tokens for the plurality of system user interface actions; and

recording the one or more system user interface tokens with the external UI recorder.

4. The computer-readable medium of claim 1 further comprising:

receiving a plurality of raw user actions for user interactions against a second graphical user interface-based application;

recording the plurality of raw user actions with the external UI recorder.

5. The computer-readable medium of claim 1 further comprising associating the one or more opaque recorded step tokens with relevant data in the external UI recorder.

6. The computer-readable medium of claim 5 wherein the relevant data comprises application identification information for the application.

7. The computer-readable medium of claim 1 wherein the one or more opaque recorded step tokens are received by the external UI recorder in batch mode from the application.

8. The computer-readable medium of claim 1 wherein the one or more opaque recorded step tokens are received by the external UI recorder in interactive mode from the application.

9. The computer-readable medium of claim 1 wherein the external UI recorder interoperates with multiple different internal macro recorders for multiple different applications, wherein the multiple different applications include the graphical user interface-based application.

10. The computer-readable medium of claim 1 wherein the method further comprises:

passing the one or more opaque recorded step tokens from the external UI recorder back to the application; and

playing back the plurality of internal macro actions.

11. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

receiving a set of one or more actions expressed in a first recording language used by a software program; and

representing the set of one or more actions as a set of one or more recorder tokens for a system UI recorder program, wherein the one or more recorder tokens encapsulate the one or more actions such that the one or more actions are opaque within the one or more recorder tokens.

12. The computer-readable medium of claim 11 wherein the software program generates the set of one or more recorder tokens.

13. The computer-readable medium of claim 11 wherein the system UI recorder program generates the set of one or more recorder tokens.

14. The computer-readable medium of claim 11 wherein the method further comprises:

receiving a second set of one or more actions expressed in a second recording language used by a second software program; and

representing the second set of or more actions as a second set of one or more recorder tokens for the system UI recorder program.

15. The computer-readable medium of claim 11 wherein the method further comprises:

receiving a second set of one or more actions expressed in a system UI recording language; and

representing the second set of one or more actions as a second set of one or more recorder tokens for the system UI recorder program.

16. The computer readable medium of claim 11 wherein the one or more recorder tokens are received by the system UI recorder program in batch mode from the software program.

17. The computer readable medium of claim 11 wherein the one or more recorder tokens are received by the system UI recorder program in interactive mode from the software program.

18. A system-wide UI recording tool for recording and playing back user interface activity, the tool comprising:

a module for collecting a first set of data associated with user interface activity against plural different applications, wherein the first set of data is expressed in any of plural native recording languages for the plural different applications;

a module for handling at least part of the collected first set of data as a set of tokens; and

a module for managing playback of the user interface activity by supplying the tokens to the respective plural different applications with which the first set of data is associated.

19. The tool of claim 18 further comprising a module for converting the set of tokens into output code

20. The tool of claim 18 wherein the modules are extensible.

21. The tool of claim 18 further comprising:

a module for collecting a second set of data associated with second user interface activity, wherein the second set of data is expressed in a non-native, system UI recording language; and

a module for managing playback of the second user interface activity by playing back the second set of data.

22. The tool of claim 18 wherein the first set of data comprises macro language instructions.

23. The tool of claim of 18 wherein the collecting the first set of data is in batch mode.

24. The tool of claim of 18 wherein the collecting the first set of data is in interactive mode.

25. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

passing a set of one or more recorder tokens from a system UI recorder program to a software program for playback by an internal recorder of the software program; and

extracting a set of one or more internal recorder actions from the set of one or more recorder tokens, wherein the set of one or more internal recorder actions is expressed in an internal recording language used by the software program, whereby the system UI recorder program interoperates with the internal recorder for the software program.

26. The computer-readable medium of claim 25 wherein the method further comprises:

passing a second set of one or more recorder tokens from the system UI recorder program to a second software program for playback by a second internal recorder of the second software program; and

extracting a second set of one or more internal recorder actions from the second set of one or more recorder tokens, wherein the second set of one or more internal recorder actions is expressed in a second internal recording language used by the second software program, whereby the system UI recorder program interoperates with the second internal recorder for the second software program.

27. The computer-readable medium of claim 25 wherein the method further comprises:

playing back a second set of one or more recorder tokens to programmatically control a second software program, wherein the second set of one or more recorder tokens is expressed in a system UI recording language, whereby the system UI recorder program interoperates with the second software program.