Cognitive control framework for automatic control of application programs exposure a graphical user interface

• US 8,401,221 B2
• Filed: 06/10/2005
• Issued: 03/19/2013
• Est. Priority Date: 06/10/2005
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Milov, Denis S., Fedorova, Julia G., Tcipnjatov, Eugene V.
• Primary Examiner(s): LEE, JOHN W
• Application Number:
  US10/576,170
• Publication Number:
  US 20090103769A1
• Time in Patent Office: 2,839 Days
• Field of Search: 382/100, 715/704
• US Class Current: 382/100
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 9/45512 Command shells

Abstract

A cognitive control framework system for automatically controlling execution of an application program having a graphical user interface includes a recording component, an execution scenario script, and a playback component. The recording component is adapted to capture user input data and images displayed by the graphical user interface during a recording phase of execution of the application program, and to analyze the captured user input data and displayed images to generate an execution scenario (script) during the recording phase. The execution scenario may be written in a selected high level language (e.g., XML). The playback component is adapted to generate simulated user input data based on the execution scenario during a playback phase of execution of the application program.

Claims

1. A computer-implemented method of automatically controlling execution of an application program by a computer system, the application program having a graphical user interface for displaying information on a display coupled to the computer system comprising:

learning an input action performed by a user while the user is using the application program and its graphical user interface during a recording phase of executing the application program by the computer system;

capturing images displayed by the graphical user interface in response to the input action being learned during the recording phase;

analyzing the user input action being learned and the resultant displayed images to generate an execution scenario script during the recording phase, wherein the learned input action and captured images are to be stored in a memory;

generating simulated user input action based on the execution scenario script during a playback phase of executing the application program by the computer system and inputting the simulated user input action to the application program;

performing image analysis on images displayed by the graphical user interface on the display as a result of processing the simulated user input action during the playback phase and captured displayed images from the recording phase;

checking recorded time conditions in the execution scenario script to handle variations in playback speed, and

automatically controlling execution of the application program by the computer system based at least in part on the image analysis.

2. The method of claim 1, wherein, performing image analysis further comprises comparing the displayed images captured during the recording phase with displayed images from the playback phase to detect an object to which the user applied the input action during the recording phase, finding the corresponding object in a displayed image from the playback phase, and applying the input action to the found corresponding object.

3. The method of claim 1, wherein the user input action comprises at least one of keyboard input action and mouse input action.

4. The method of claim 3, wherein learning the input action during the recording phase comprises registering and capturing the input action performed by the user.

5. The method of claim 1, wherein the execution scenario script comprises a script including extended markup language (XML) tags.

6. The method of claim 1, wherein analyzing the captured user input data and displayed images during a recording phase comprises:

determining contours of objects shown in a displayed image;

detecting an object of activity from among the objects; and

detecting additional objects located adjacent to the object of activity.

7. The method of claim 1, wherein performing image analysis comprises determining contours of objects in an image displayed by the graphical user interface during the playback phase and filtering the objects according to size to produce a set of hypotheses for active objects.

8. The method of claim 7, wherein performing image analysis further comprises filtering objects according to space layout to produce a first subset of hypotheses for active objects.

9. The method of claim 7, wherein performing image analysis further comprises filtering objects by content to produce a second subset of hypotheses for active objects.

10. The method of claim 7, wherein performing image analysis further comprises structural filtering of objects to produce a best hypothesis for an active object.

11. The method of claim 7, wherein performing image analysis further comprises recalculating old actions for a new object identified as the active object.

12. A non-transitory computer readable medium comprising instructions, which when executed, result in automatically controlling execution of an application program by a computer system, the application program having a graphical user interface for displaying information on a display coupled to the computer system by

registering and capturing an input action performed by a user while the user is using the application program and its graphical user interface during a recording phase of executing the application program by the computer system;

capturing images displayed by the graphical user interface in response to the input action performed by the user during the recording phase of execution of the application program by the computer system;

analyzing the registered and captured user input action and responsively displayed images to generate an execution scenario script during the recording phase;

generating a simulated user input action based on the execution scenario script during a playback phase of execution of the application program by the computer system and inputting the simulated user input action to the application program;

performing image analysis on images displayed by the graphical user interface on the display as a result of processing the simulated user input action during the playback phase and captured displayed images from the recording phase;

checking recorded time conditions in the execution scenario script to handle variations in playback speed, and

automatically controlling execution of the application program by the computer system based at least in part on the image analysis.

13. The non-transitory computer readable medium of claim 12, wherein instructions for performing image analysis comprise instructions for comparing the displayed images captured during the recording phase with displayed images from the playback phase.

14. The non-transitory computer readable medium of claim 12, wherein the user input action comprises at least one of keyboard input action and mouse input action.

15. The non-transitory computer readable medium of claim 12, wherein instructions for analyzing the registered and captured user input action and responsively displayed images in the recording phase comprise instructions for identifying actions of the application program corresponding to the registered and captured user input action.

16. The non-transitory computer readable medium of claim 12, wherein the execution scenario script comprises a script including extended markup language (XML) tags.

17. The non-transitory computer readable medium of claim 12, wherein instructions for analyzing the captured user input data and displayed images during a recording phase comprise instructions for:

determining contours of objects shown in a displayed image;

detecting an object of activity from among the objects; and

detecting additional objects located adjacent to the object of activity.

18. The non-transitory computer readable medium of claim 12, wherein instructions for performing image analysis comprise instructions for determining contours of objects in an image displayed by the graphical user interface during the playback phase and filtering the objects according to size to produce a set of hypotheses for active objects.

19. The non-transitory computer readable medium of claim 18, wherein instructions for performing image analysis further comprise instructions for filtering objects according to space layout to produce a first subset of hypotheses for active objects.

20. The non-transitory computer readable medium of claim 18, wherein instructions for performing image analysis further comprise instructions for filtering objects by content to produce a second subset of hypotheses for active objects.

21. The non-transitory computer readable medium of claim 18, wherein instructions for performing image analysis further comprise instructions for structural filtering of objects to produce a best hypothesis for an active object.

22. The non-transitory computer readable medium of claim 18, wherein instructions for performing image analysis further comprise instructions for recalculating old actions for a new object identified as the active object.

23. A cognitive control framework system for automatically controlling execution of an application program having a graphical user interface in a computer system comprising:

a recording component adapted to register and capture user input actions and capture images displayed by the graphical user interface as a result of the user input actions during a recording phase of execution of the application program, and to analyze the registered and captured user input actions and captured resultant displayed images to generate an execution scenario script during the recording phase,

a playback component adapted to generate simulated user input actions based on the execution scenario script during a playback phase of execution of the application program, to input the simulated user input actions to the application program, to perform image analysis on images displayed by the graphical user interface as a result of processing the simulated user input actions during the playback phase and captured displayed images from the recording phase;

checking recorded time conditions in the execution scenario script to handle variations in playback speed, and to automatically control execution of the application program by the computer system based at least in part on the image analysis.

24. The system of claim 23, wherein the playback component is adapted to compare the displayed images captured during the recording phase with displayed images from the playback phase.

25. The system of claim 23, wherein the recording component is adapted to identify actions of the application program corresponding to the registered and captured user input actions.