Method and apparatus for analyzing computer screens

• US 6,226,407 B1
• Filed: 03/18/1998
• Issued: 05/01/2001
• Est. Priority Date: 03/18/1998
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zabih, Ramin, Halviatti, Ramin
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US09/044,209
• Time in Patent Office: 1,140 Days
• Field of Search: 382/154, 382/209, 382/218, 382/219, 382/305, 382/309, 382/180, 707/513, 707/516
• US Class Current: 382/209
• CPC Class Codes: G06T 2207/30176 Document
• CPC Class Codes: G06T 7/001 using an image reference approach

Abstract

An invention for quantitatively analyzing bitmap images using computer vision is disclosed. In one embodiment, these bitmap images correspond to the rendered screens of two applications based on a same data source file. An optional preprocessing step filters non-important information for analyzing purposes such as the background, replaces each image with a blank image of the same size, and modifies non-visible attributes of a screen so their structure can be recovered during the computer vision processing. After a series of rendered screens of each application are captured, these bitmap images are interpreted using computer vision techniques to produce machine-readable visual attributes of the rendered screens. Corresponding attributes from each of the two applications are then compared to generate a set of differences, where those differences within a predefined set of deliberate design differences are ignored. These attributes and differences are then processed to derive a set of grades reflecting the similarities between the rendered screens of the two applications. In determining these grades, a set of user-definable thresholds are used to overlook small variances inherent in the rendering by the different applications.

Claims

1. In a computer system, a method for determining differences between first and second screen images generated by different computer applications, the method comprising the steps of:

capturing imaging data into a first image file for the fist screen image, where the imaging data has been generated by a first computer application for rendering the first screen image from machine-readable source data;

capturing imaging data into a second image file for the second screen image, where the imaging data has been generated by a second computer application, that is different from the first computer application, for rendering the second screen image from the same machine-readable source data that generated the imaging data for the first screen image;

automatically comparing the imaging data in the first and second image files; and

generating from the comparison of the image files a data structure containing visual differences between the first and second screen images.

2. The method of claim 1, wherein the imaging data generated by first and second computer applications includes marks, lines and boxes.

3. The method of claim 1, further comprising the step of massaging a data source to generate the machine-readable source data.

4. The method of claim 3, wherein the data source is a HTML file.

5. The method of claim 3, wherein the step of massaging the data source includes modifying a background image or a picture associated with the data source.

6. The method of claim 1, wherein the first and second screen images include concatenated bitmap captured screen images;

the step of generating imaging data for the first screen image includes running the first computer application; and

the step of generating imaging data for the second screen image includes running the second computer application.

7. The method of claim 1, wherein generating the data structure containing visual differences between the first and second screen images, further comprises referencing a set of deliberate differences.

8. The method of claim 7, wherein one or more elements of the set of deliberate differences are run-length encoded.

9. The method of claim 1, further comprising the step of grading the visual differences between the first and second screen images.

10. The method of claim 9, wherein the step of grading references a set of threshold values for determining whether or not to recognize a particular difference between the first and second screen images.

11. The method of claim 1, further comprising the step of visually displaying the data structure containing visual differences between the first and second screen images.

12. The method of claim 11, wherein the step of generating the data structure containing visual differences between the first and second screen image references a set of threshold values for determining whether or not to recognize a particular difference between the first and second screen images;

and the method further comprising the step of modifying one or more threshold values within the set of threshold values in response to user input.

13. A computer-readable medium having computer-executable instructions for performing the steps recited in claim 1.

14. In a computer system, an automated method for detecting differences between first and second screen images produced by different versions of a computer application, the method comprising the steps of:

interpreting the first screen image to generate a first set of machine-readable attributes describing visual attributes of the first screen image;

interpreting the second screen image to generate a second set of machine-readable attributes describing visual attributes of the second screen image; and

automatically processing the first and second sets of machine-readable attributes to provide a comparison between the first and second screen images.

15. In a computer system, a method for comparing first and second digital images respectively corresponding to first and second outputs of first and second different computer applications, the method comprising the steps of:

populating a first data structure with a set of machine-readable attributes describing the first digital image by determining related picture elements (pixels) comprising components of the first digital image;

populating a second data structure with a set of machine-readable attributes describing the second digital image by determining related pixels comprising components of the second digital image; and

analyzing the machine-readable attributes represented in the first and second data structures to automatically generate a comparison of the first and second digital images.

16. A computer-readable medium having computer-executable instructions for performing the steps recited in claim 15.

17. The method of claim 15, wherein the step of analyzing the machine-readable attributes further comprises referencing a set of deliberate differences.

18. The method of claim 17, wherein one or more elements of the set of deliberate differences are run-length encoded.

19. The method of claim 15, wherein the steps of populating the first and second data structures include executing a connected points means.

20. The method of claim 15, wherein the first and second computer applications use the same source data in producing the first and second outputs.

21. The method of claim 15, wherein the machine-readable attributes include marks, lines and boxes.

22. The method of claim 15, wherein the comparison includes a rendering grade indicating how many marks from the first data structure are similar or identical to marks from the second data structure.

23. The method of claim 15, wherein the comparison includes a line break grade indicating the number of lines from the first data structure having the same number of glyphs as corresponding lines from the second data structure.

24. The method of claim 15, wherein the comparison includes an alignment grade indicating the number of lines from the first data structure having the same horizontal alignment as corresponding lines from the second data structure.

25. The method of claim 15, wherein the comparison includes a spacing grade indicating the number of lines from the first data structure having the same line spacing as corresponding lines from the second data structure.

26. The method of claim 15, wherein the comparison includes a placement grade indicating the similarity of character placement within corresponding lines in the first and second data structures.

27. The method of claim 15, wherein the comparison includes a size grade indicating the similarity in the size of a box in the first data structure with a box in the second data structure.

28. The method of claim 15, wherein the analyzing of the machine-readable attributes includes referencing a set of predetermined threshold values for determining whether or not to recognize a particular difference between the first and second digital images.

29. The method of claim 28, wherein the predetermined threshold values are set on a per digital image basis.

30. A computer-readable medium having instructions for determining differences between first and second screen images generated by first and second computer applications respectively, the instructions performing the steps of:

capturing imaging data into a first image file for the first screen image, where the imaging data has been generated by a first computer application for rendering the first screen image from machine-readable source data;

capturing imaging data into a second image file for the second screen image, where the imaging data has been generated by a second computer application, that is different from the first computer application, for rendering the second screen image from the same machine-readable source data that generated the imaging data for the first screen image;

automatically comparing the imaging data in the first and second image files; and

generating from the comparison of the image files a data structure containing visual differences between the first and second screen images.

31. The method of claim 30, further comprising the step of grading the visual differences between the first and second screen images.