Graphical user interface (GUI) noise reduction in a cognitive control framework

• US 8,429,612 B2
• Filed: 11/11/2005
• Issued: 04/23/2013
• Est. Priority Date: 11/11/2005
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Milov, Denis Sergeevich
• Primary Examiner(s): Do, Chat
• Assistant Examiner(s): LOUIE, JUE WANG
• Application Number:
  US10/587,342
• Publication Number:
  US 20080294985A1
• Time in Patent Office: 2,720 Days
• Field of Search: None
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Reducing graphical user interface (GUI) noise maybe achieved by recording a first execution scenario for control of operation of an application program having a GUI during a recording phase of operation of a cognitive control framework system, setting soft conditions for a search for the application program for the first execution scenario, playing back the application program according to the first execution scenario during a playback phase of operation of the cognitive control framework system, updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during playback, including updating recorded images originally generated by the GUI during the recording phase and updating coordinates for user input data, setting stronger conditions for the search for use in subsequent playbacks; and playing back the application program according to the second execution scenario with the stronger conditions for search.

Claims

1. A computer-implemented method of reducing graphical user interface (GUI) noise comprising:

during a recording phase of operation of a cognitive control framework system, capturing and storing user input actions and corresponding displayed images responsive to individual input actions while the user is using a graphical user interface (GUI) of an application program, and creating a first execution scenario based on the captured user input actions and corresponding displayed images;

during a first playback phase of operation of the cognitive control framework system, automatically providing the captured user input actions of the first execution scenario to the application program, for each of the captured user input actions automatically provided to the application program, generating and capturing a first playback image responsive to the provided captured user input action, performing image analysis on the generated first playback image using a recorded image of the first execution scenario displayed responsive to a next provided captured user input action, finding a next object to receive the next provided captured user input action in the generated first playback image based on the image analysis, the finding of the next object based on at least one search parameter having a broad limit to enable the next object to be found in spite of a noise-induced difference between the generated first playback image and the recorded image of the first execution scenario, and recalculating input action data for the next provided captured user input action according to the found next object;

modifying the first execution scenario to form a second execution scenario to reduce the noise-induced differences observed in the GUI during the first playback phase, the modifications to include the recalculated input action data and the first playback images;

resetting the at least one search parameter to have a narrow limit compared to the broad limit, the narrow limit to account for the reduction in the noise-induced differences in the GUI; and

playing back the application program according to the second execution scenario, displaying a second playback image in response to a recalculated input action from the second execution scenario, performing image analysis on the second playback image to find the next object in the second playback image, the image analysis performed using a first playback image in the second execution scenario corresponding to the next recalculated input action from the second execution scenario, the finding of the next object in the second playback image based on the at least one search parameter having the narrow limit.

2. The method of claim 1, wherein the at least one search parameter having a broad limit is to account for noise-induced differences that occur in at least one of shapes of contours, text and image content, or layout.

3. The method of claim 2, where the at least one search parameter is reset to have a narrow limit with respect to differences that occur in at least one of shapes of contours, text and image content, or layout.

4. The method of claim 1, wherein a user input action recorded during the recording phase comprises a mouse selection, and the first execution scenario is modified to include recalculated display coordinates to indicate where the mouse selection is to be automatically applied according to the second execution scenario, the display coordinates being recalculated based on the automatic mouse selection of a next object found in a first playback image.

5. The method of claim 1, wherein GUI noise comprises at least one of changeable color schemes, highlighting of items, noise from video sources, and anti-aliasing effects.

6. The method of claim 5, further comprising:

during the recording phase, highlighting a display item in response to a given user input action and capturing the given user input action and the corresponding displayed image having the highlighted display item;

during the first playback phase, comparing a currently generated first playback image responsive to a prior captured user input action to the recorded displayed image having the highlighted display item, the highlighting comprising a noise-induced difference between the currently generated first playback image and the recorded displayed image, based on the comparison and the at least one search parameter having a broad limit, finding the next object in the currently generated first playback image that is to receive the captured given user input action, and in response to providing the captured given user input action to the application program, generate a subsequent first playback image with the found next object as a highlighted display item.

7. An article comprising:

a non-transitory machine accessible storage medium containing instructions, which when executed, result in reducing graphical user interface (GUI) noise by, during a recording phase of operation of a cognitive control framework system, capturing and storing user input actions and displayed images responsive to the user input actions while the user is using a graphical user interface (GUI) of an application program, and creating a first execution scenario based on the captured user input actions and the displayed images;

during a first playback phase of operation of the cognitive control framework system, automatically providing the captured user input actions of the first execution scenario to the application program, generating and capturing a first playback image responsive to a particular provided captured user input action, finding an object in the generated first playback image which is to receive a next provided captured user input action, the finding of the object based on image analysis of the generated first playback image using a recorded image of the first execution scenario displayed responsive to the next captured user input action and on at least one search parameter having a broad limit to enable the object to be found in spite of a noise-induced difference between the generated first playback image and the recorded image of the first execution scenario, and recalculating input action data for the provided next captured user input action based on the found object;

updating the first execution scenario to form a second execution scenario to reduce the noise-induced differences observed in the GUI during the first playback phase, the updates to include the recalculated input action data and the first playback images;

resetting the at least one search parameter to have a narrow limit compared to the broad limit, the narrow limit to account for the reduction in the noise-induced differences in the GUI; and

playing back the application program according to the second execution scenario, displaying a second playback image in response to a recalculated input action from the second execution scenario, performing image analysis on the second playback image to find the object in the second playback image, the image analysis performed using a first playback image from the second execution scenario which was responsive to the next recalculated input action from the second execution scenario, the finding of the object in the second playback image based on the at least one search parameter having the narrow limit.

8. The article of claim 7, wherein the at least one search parameter having a broad limit is to account for noise-induced differences that occur in at least one of shapes of contours, text and image content, or layout.

9. The article of claim 8, where the at least one search parameter is reset to have a narrow limit with respect to differences that occur in at least one of shapes of contours, text and image content, or layout.

10. The article of claim 7, wherein the particular provided captured user input action comprises a mouse selection, and the second execution scenario includes recalculated display coordinates to indicate where to apply the mouse selections in a previously displayed second playback image.

11. The article of claim 7, wherein GUI noise comprises at least one of changeable color schemes, highlighting of items, noise from video sources, and anti-aliasing effects.

12. A method of automatically controlling execution of an application program having a GUI to reduce GUI noise comprising:

capturing user input data and images displayed by the GUI responsive to the captured user input data during a recording phase of execution of the application program;

analyzing the captured user input data and responsive recorded images to generate a first execution scenario during the recording phase;

generating simulated user input data based on the first execution scenario during a first playback phase of execution of the application program and inputting the simulated user input data to the application program;

performing image analysis on first playback images displayed by the GUI as a result of processing the simulated user input data during the first playback phase and the recorded images, the image analysis to enable finding, in each first playback image, a next object to receive a next input action according to the first execution scenario, the finding based on a search parameter having a broad limit to enable the next objects to be found in spite of GUI noise conditions that create differences between the recorded images and the first playback images;

updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during the first playback phase, the updates including the first playback images and recalculated user input data corresponding to the next objects found in the first playback images;

resetting the search parameter to have a narrow limit as compared to the broad limit to account for the reduced GUI noise; and

playing back the application program according to the second execution scenario and performing image analysis on second playback images displayed by the GUI as a result of processing simulated user input data and first playback images in the second execution scenario to find the next object in each second playback image, the finding of the next object in the second play back images based at least on the reset search parameter having the narrow limit.

13. The method of claim 12, wherein the search parameter having a broad limit comprises a first set of bounds for differences in one of shapes of contours, text and image content, or layout.

14. The method of claim 13, where the search parameter having the narrow limit comprises a second set of bounds for differences in one of shapes of contours, text and image content, or layout, the second set being different than the first set.

15. The method of claim 13, wherein GUI noise conditions comprise at least one of changeable color schemes, highlighting of items, noise from video sources, and anti-aliasing effects.

16. An article comprising:

a non-transitory machine accessible storage medium containing instructions, which when executed, result in automatically controlling execution of an application program having a GUI to reduce GUI noise by, capturing user input data and images displayed by the GUI responsive to the captured user input data during a recording phase of execution of the application program;

analyzing the captured user input data and responsive recorded images to generate a first execution scenario during the recording phase;

generating simulated user input data based on the first execution scenario during a first playback phase of execution of the application program and inputting the simulated user input data to the application program;

performing image analysis on first playback images displayed by the GUI as a result of processing the simulated user input data during the first playback phase and the recorded images, the image analysis to enable finding, in each first playback image, a next object to receive a next input action according to the first execution scenario, the finding based on a search parameter having a broad limit to enable the next objects to be found in spite of GUI noise conditions that create differences between the recorded images and the first playback images;

updating the first execution scenario to form a second execution scenario to reduce GUI noise conditions observed during the first playback phase, the updates including the first playback images and recalculated user input data corresponding to the next objects found in the first playback images;

resetting the search parameter to have a narrow limit as compared to the broad limit to account for the reduced GUI noise; and

playing back the application program according to the second execution scenario and performing image analysis on second playback images displayed by the GUI as a result of processing simulated user input data and first sla back ima es in the second execution scenario to find the next object in each second playback image, the finding of the next object in the second play back images based at least on the reset search parameter having the narrow limit.

17. The article of claim 16, wherein the search parameter having a broad limit comprises a first set of bounds for differences in one of shapes of contours, text and image content, or layout.

18. The article of claim 17, where the search parameter having the narrow limit comprises a second set of bounds for differences in one of shapes of contours, text and image content, or layout, the second set being different than the first set.

19. The article of claim 16, wherein GUI noise conditions comprise at least one of changeable color schemes, highlighting of items, noise from video sources, and anti-aliasing effects.