Smart UI recording and playback framework

• US 7,653,896 B2
• Filed: 06/30/2004
• Issued: 01/26/2010
• Est. Priority Date: 06/30/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Herdeg, Howard B. III
• Primary Examiner(s): Dam; Tuan Q
• Assistant Examiner(s): TECKLU, ISAAC TUKU
• Application Number:
  US10/882,861
• Publication Number:
  US 20060005132A1
• Time in Patent Office: 2,036 Days
• Field of Search: 717/105, 717/106
• US Class Current: 717/113
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 9/45512 Command shells

Abstract

Techniques and tools are described for recording and reliably replicating graphical user interface (“GUI”) activity for various applications, including applications that have dynamic user interfaces. The techniques and tools may be used in combination or separately. For example, a recording tool receives internal macro data from a GUI-based application as opaque tokens, which are embedded into playback code. A playback tool executes the playback code by passing the embedded tokens back to their original application for playback by the application'"'"'s internal recorder.

Claims

1. A computer-readable storage medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

receiving a plurality of internal macro actions passed as one or more opaque recorded step tokens from a graphical user interface-based application across a native recording interface to an external UI recorder configurable to interoperate with multiple different internal macro recorders for multiple different applications, including the graphical user interface-based application, the internal macro actions being recorded in a native format for the graphical user interface-based application;

recording the one or more opaque recorded step tokens with the external UI recorder;

associating the one or more opaque recorded step tokens with relevant data in the external UI recorder; and

generating playback code, including;

mapping at least one logical recording event token into corresponding instructions in the playback code; and

combining into a single code stream the one or more opaque recorded step tokens and the corresponding instructions for the at least one logical recording event token, wherein the one or more opauue recorded step tokens are embedded in the playback code without conversion to other expressions in the playback code;

wherein the one or more opaque recorded step tokens comprise discrete chunks of data whose contents are not interpreted by the external UI recorder but are recognizable in the graphical user interface-based application.

2. The computer-readable storage medium of claim 1 wherein the plurality of internal macro actions are generated by an internal macro recorder of the graphical user interface-based application and wherein the one or more opaque recorded step tokens are for handling by the external UI recorder to interoperate with the internal macro recorder of the graphical user interface-based application.

3. The computer-readable storage medium of claim 1 wherein the method further comprises:

receiving a plurality of system user interface actions for user interactions against a second graphical user interface-based application;

generating one or more system user interface tokens for the plurality of system user interface actions; and

recording the one or more system user interface tokens with the external UI recorder.

4. The computer-readable storage medium of claim 1 wherein the method further comprises:

receiving a plurality of raw user actions for user interactions against a second graphical user interface-based application; and

recording the plurality of raw user actions with the external UI recorder.

5. The computer-readable storage medium of claim 1 wherein the relevant data comprises application identification information for the graphical user interface-based application.

6. The computer-readable storage medium of claim 1 wherein the one or more opaque recorded step tokens are received by the external UI recorder in batch mode from the graphical user interface-based application.

7. The computer-readable storage medium of claim 1 wherein the one or more opaque recorded step tokens are received by the external UI recorder in interactive mode from the graphical user interface-based application.

8. The computer-readable storage medium of claim 1 wherein the method further comprises:

passing the one or more opaque recorded step tokens from the external UI recorder back to the graphical user interface-based application; and

playing back the plurality of internal macro actions.

9. The computer-readable storage medium of claim 1 wherein the relevant data comprises timestamps and index values so that the recorded step tokens can be played back in proper sequence.

10. The computer-readable storage medium of claim 1 wherein at least some of the one or more opaque recorded step tokens comprise parameters of a method call.

11. The computer-readable storage medium of claim 1 wherein at least some of the one or more opaque recorded step tokens comprise a persistent ID associated with a UT element that allows the UT element to be uniquely identified in another instance of the graphical user interface-based application.

12. The computer-readable storage medium of claim 11 wherein at least some of the one or more opaque recorded step tokens farther comprise hierarchical identification information and control-pattern specific information.

13. The computer-readable storage medium of claim 1 wherein the method farther comprises the external UT recorder receiving notification from the graphical user interface-based application that the graphical user interface-based application is compatible with the external UT recorder.

14. The computer-readable storage medium of claim 1 wherein the method farther comprises the external UT recorder collecting input data from a graphical user interface, resolving the user input data to at least one user interface element affected by the user input data, and expressing results of the resolving as the at least one logical recording event token.

15. In a computing device that implements an application having a native macro recording environment, a method of tokenizing user interactions and passing the tokenized user interactions to a system UI recorder program, the method comprising:

receiving input that indicates user interactions with the application having the native macro recording environment;

with the computing device that implements the application, recording the user interactions in the application'"'"'s native macro recording environment;

with the computing device that implements the application, representing the user interactions as one or more recorder tokens for a system UI recorder program configured to interoperate with multiple different internal macro recorders for multiple different applications, including the application having the native macro recording environment, wherein the one or more recorder tokens encapsulate the user interactions such that the one or more recorder tokens are macro language independent but contents of the one or more recorder tokens are recognizable to the application'"'"'s native macro recording environment, and wherein the one or more recorder tokens are passed as opaque chunks of data whose contents are not interpreted by the system UI recorder program;

with the computing device that implements the application, passing the one or more recorder tokens to the system UI recorder program for generation of playback code that includes;

mapping at least one logical recording event token into corresponding instructions in the playback code; and

combining into a single code stream the one or more recorder tokens, whose contents are recognizable to the application'"'"'s native macro recording environment, and the corresponding instructions for the at least one logical recording event token, wherein the one or more recorder tokens are embedded in the playback code without conversion to other expressions in the playback code;

such that the user interactions are opaque within the one or more recorder tokens to the system UI recorder program.

16. The method of claim 15 further comprising:

with the computing device that implements the application, receiving a second set of one or more actions expressed in a system UI recording language; and

with the computing device that implements the application, representing the second set of one or more actions as the at least one logical recording event token.

17. The method of claim 15 wherein the one or more recorder tokens are received by the system UI recorder program in batch mode from the application.

18. The method of claim 15 wherein the one or more recorder tokens are received by the system UI recorder program in interactive mode from the application.

19. The method of claim 15 wherein the application represents at least one user interaction at a location as at least one macro language instruction that hooks into an internal object model of the application, and wherein at least one recorder token encapsulates the at least one macro language instruction to facilitate playback despite changes in interface layout of a user interface in which the macro language instruction is played back.

20. The method of claim 15 further comprising:

receiving input that indicates user interactions with a second of the plural applications having a second native macro recording environment;

with the computing device that implements the application, recording the user interactions with the second application in the second native macro recording environment;

with the computing device that implements the application, representing the user interactions with the second application as a second set of one or more recorder tokens for the system UI recorder program, wherein the second set of one or more recorder tokens encapsulate the user interactions with the second application such that the second set of one or more recorder tokens are macro language independent but contents of the second set of one or more recorder tokens are recognizable to the second application'"'"'s native macro recording environment; and

with the computing device that implements the application, passing the second set of one or more recorder tokens to the system UI recorder program for combination into the single code stream, such that the user interactions with the second application are opaque within the second set of one or more recorder tokens to the system UI recorder program.

21. A computer system comprising:

computer memory and a central processing unit;

a system-wide UI recording tool stored in computer memory and executable using the central processing unit for recording and playing back user interface activity, the system-wide UI recording tool comprising;

a module for collecting a first set of data associated with user interface activity against plural different applications, wherein the first set of data is expressed in any of plural native recording languages for the plural different applications;

a module for storing the collected first set of data in memory associated with the system-wide UI recording tool;

a module for retrieving at least a portion of the stored collected first set of data from memory and for converting the at least a portion of the stored collected first set of data into a set of tokens, the converting including adding marking data to the set of tokens, the marking information comprising the application name and a timestamp;

a module for verifying the set of tokens as internally consistent, the verifying comprising considering marking information added to the set of tokens;

wherein the set of tokens are passed as opague chunks of data whose contents are not interpreted by the system-wide UI recording tool;

a module for generating playback code, wherein the generating includes;

mapping at least one logical recording event token into corresponding instructions in the playback code; and

combining into a single code stream the set of tokens and the corresponding instructions for the at least one logical recording event token, wherein the set of tokens are embedded in the playback code without conversion to other expressions in the playback code; and

a module for managing playback of the user interface activity by supplying at least some of the set of tokens to the respective plural different applications with which the first set of data is associated, the respective plural different applications determined using the respective plural application names found in the set of tokens.

22. The computer system of claim 21 wherein the modules are extensible.

23. The computer system of claim 21 wherein the system-wide UI recording tool further comprises:

a module for collecting a second set of data associated with second user interface activity, wherein the second set of data is expressed in a non-native, system UI recording language, and wherein the at least one logical recording event token represents the second set of data expressed in the non-native, system UI recording language.

24. The computer system of claim 21 wherein the first set of data comprises macro language instructions.

25. The computer system of claim 21 wherein the collecting the first set of data is in batch mode.

26. The computer system of claim 21 wherein the collecting the first set of data is in interactive mode.

27. The computer system of claim 21 wherein the system-wide UI recording tool provides a heavyweight recording mode which collects detailed information about the user interface activity, and a lightweight recording mode which collects less information about the user interface activity than the heavyweight recording mode.

28. In a computing device that implements a software program with an internal recorder and native playback interface, a method comprising:

passing a set of one or more recorder tokens from a system UI recorder program to the software program for playback by the internal recorder of the software program using the native playback interface, the one or more recorder tokens being passed as opaque chunks of data whose contents are not interpreted by the system UI recorder program, wherein the system UI recorder program is configurable to interoperate with multiple different internal recorders for multiple software programs, including the software program having the native playback interface, and wherein the set of one or more recorder tokens are embedded in playback code generated by;

mapping at least one logical recording event token into corresponding instructions in the playback code; and

combining into a single code stream the set of one or more recorder tokens and the corresponding instructions for the at least one logical recording event token, wherein the set of one or more recorder tokens are embedded in the playback code without conversion to other expressions in the playback code;

with the computing device that implements the software program, extracting a set of one or more internal recorder actions from the set of one or more recorder tokens, wherein the set of one or more internal recorder actions is expressed in an internal recording language used by the software program, whereby the system UI recorder program interoperates with the internal recorder for the software program;

processing marking data associated with at least one recorder token of the set of one or more recorder tokens, the marking data comprising application information, process information, and at least one timestamp;

with the computing device that implements the software program, executing a recorded action associated with the at least one recorder token; and

with the computing device that implements the software programs, reporting results of the recorded action.

29. The method of claim 28 wherein the method further comprises:

passing a second set of one or more recorder tokens from the system UI recorder program to a second software program of the multiple software programs, for playback by a second internal recorder of the second software program; and

extracting a second set of one or more internal recorder actions from the second set of one or more recorder tokens, wherein the second set of one or more internal recorder actions is expressed in a second internal recording language used by the second software program, whereby the system UI recorder program interoperates with the second internal recorder for the second software program.

30. The method of claim 28 wherein the method further comprises:

playing back the corresnonding instructions to programmatically control a second software program of the multiple software programs, the at least one logical recording event token representing system UI actions expressed in a system UI recording language, whereby the system UI recorder program interoperates with the second software program.

31. The method of claim 28 wherein the method further comprises, from the system UI recorder program to the software program, passing the corresponding instructions generated from the at least one logical recording event token, which represent UI system architecture inputs, for programmatic playback.