COGNITIVE CONTROL FRAMEWORK FOR AUTOMATIC CONTROL OF APPLICATION PROGRAMS EXPOSURE A GRAPHICAL USER INTERFACE
First Claim
1. A method of automatically controlling execution of an application program having a graphical user interface comprising:
- capturing user input data and images displayed by the graphical user interface during a recording phase of execution of the application program;
analyzing the captured user input data and displayed images to generate an execution scenario during the recording phase;
generating simulated user input data based on the execution scenario during a playback phase of execution of the application program and inputting the simulated user input data to the application program;
performing image analysis on images displayed by the graphical user interface as a result of processing the simulated user input data during the playback phase and captured displayed images from the recording phase; and
automatically controlling execution of the application program based at least in part on the image analysis.
1 Assignment
0 Petitions
Accused Products
Abstract
The present invention relates generally to automatic control of software application programs and image analysis and, more specifically, to analyzing graphical component, an execution scenario script, and a playback component. The recording component is adapted to capture user input data and images displayed by the graphical user interface during a recording phase of execution of the application program, and to analyze the captured user input data and displayed images to generate an execution scenario (script) during the recording phase. The execution scenario may be written in a selected high level language (e.g., XML). The playback component is adapted to generate simulated user input data based on the execution scenario during a playback phase of execution of the application program, to input the simulated user input data to the application program, to perform image analysis on images displayed by the graphical user interface as a result of processing the simulated user input data during the playback phase and captured displayed images from the recording phase; and to automatically control execution of the application program based at least in part on the image analysis.
68 Citations
30 Claims
-
1. A method of automatically controlling execution of an application program having a graphical user interface comprising:
-
capturing user input data and images displayed by the graphical user interface during a recording phase of execution of the application program; analyzing the captured user input data and displayed images to generate an execution scenario during the recording phase; generating simulated user input data based on the execution scenario during a playback phase of execution of the application program and inputting the simulated user input data to the application program; performing image analysis on images displayed by the graphical user interface as a result of processing the simulated user input data during the playback phase and captured displayed images from the recording phase; and automatically controlling execution of the application program based at least in part on the image analysis. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. An article comprising:
- a machine readable medium containing instructions, which when executed, result in automatically controlling execution of an application program having a graphical user interface by capturing user input data and images displayed by the graphical user interface during a recording phase of execution of the application program;
analyzing the captured user input data and displayed images to generate an execution scenario during the recording phase; generating simulated user input data based on the execution scenario during a playback phase of execution of the application program and inputting the simulated user input data to the application program; performing image analysis on images displayed by the graphical user interface as a result of processing the simulated user input data during the playback phase and captured displayed images from the recording phase; and automatically controlling execution of the application program based at least in part on the image analysis. - View Dependent Claims (13, 14, 15, 16, 17, 18, 19, 20, 21, 22)
- a machine readable medium containing instructions, which when executed, result in automatically controlling execution of an application program having a graphical user interface by capturing user input data and images displayed by the graphical user interface during a recording phase of execution of the application program;
-
23. A cognitive control framework system for automatically controlling execution of an application program having a graphical user interface comprising:
-
a recording component adapted to capture user input data and images displayed by the graphical user interface during a recording phase of execution of the application program, and to analyze the captured user input data and displayed images to generate an execution scenario during the recording phase, a playback component adapted to generate simulated user input data based on the execution scenario during a playback phase of execution of the application program, to input the simulated user input data to the application program, to perform image analysis on images displayed by the graphical user interface as a result of processing the simulated user input data during the playback phase and captured displayed images from the recording phase; and
to automatically control execution of the application program based at least in part on the image analysis. - View Dependent Claims (24, 25, 26, 27, 28, 29, 30)
-
Specification