Application testing with virtual object recognition

• US 6,046,740 A
• Filed: 02/06/1998
• Issued: 04/04/2000
• Est. Priority Date: 02/07/1997
• Status: Expired due to Term

• Current Assignee: Borland Software Corporation (Open Text Corporation)
• Original Assignee: Segue Software, Inc. (Open Text Corporation)
• Inventors: Anderson, Timothy A., LaRoche, David C.
• Primary Examiner(s): Bayerl, Raymond J.
• Assistant Examiner(s): Luu, Sy D.
• Application Number:
  US09/020,011
• Time in Patent Office: 788 Days
• Field of Search: 345/339, 345/333, 345/335, 345/346, 345/356, 345/967, 345/433, 345/441, 345/443, 382/195, 382/197, 382/203, 382/225, 382/226, 714/38
• US Class Current: 715/764
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06V 30/422 Technical drawings; Geographical maps
• CPC Class Codes: Y10S 715/967 Visual or iconic programming

Abstract

A method of recognizing graphical objects by subjecting the graphical information gathered through "spying" to a series of rules by which the object becomes understood or recognized as an instance of a standard logical object. Before the rules are applied, graphical objects are first interpreted as primitives including groups of text, lines and images. In order to recognize a graphical object as a logical object, the graphical information is subjected to the rules in an iterative process whereby an understanding of the object is continually refined. As the rules are applied, the results are evaluated to determine whether the graphical object can be "mapped" to a standard logical object such as a textfield or listbox. Once the object is understood as a logical element with which the user is accustomed, it is possible to interact with the object and obtain data from the object as if it were a standard object with a published interface. By subjecting the graphical data to a series of rules designed specifically to recognize tables, the boundaries and the internal structure of rows and columns will be understood. Once the graphical data is recognized as a table, the data which it contains in rows can then be accessed. By classifying an object as an instance of a known object, assumptions can be made about the object so that it can be navigated or validated by sending events or messages.

Claims

1. A method for recognizing graphical objects comprising the following steps:

obtaining a current display state from a graphical user interface operating system;

classifying displayed windows as canvas windows or control windows;

classifying control windows which are separate windows as native controls, otherwise classifying control windows which are part of a canvas window as rendered controls;

transforming a real window hierarchy which is used to create the displayed windows into a perceived window hierarchy corresponding to the visual display of the displayed windows;

identifying graphical objects in the perceived window hierarchy as graphical object primitives;

applying a series of rules to the graphical object primitives to characterize the graphical object primitives as logical objects based on visual characteristics of said graphical object primitives.

2. The method of claim 1, further comprising the step of updating the perceived window hierarchy as windows are created or destroyed.

3. The method of claim 1, wherein the perceived window hierarchy includes container windows and wherein each of said container windows is capable of including child windows.

4. The method of claim 1, further comprising the step of synchronizing access to data structures by blocking execution of a process thread if it requires access to a data structure already in use in another process thread.

5. The method of claim 1, further comprising the following steps:

grouping the graphical primitive objects in a window using an object grouping process thread in order to create virtual objects;

preventing an application under test from drawing while said object grouping is being performed, said preventing step comprising at least one of the following steps;

marking said object grouping process thread as time critical;

allowing said object grouping process thread to obtain a read lock on objects being grouped; and

determining whether the application under test is in a ready condition.

6. The method of claim 5, further comprising the step of adding at least one new graphical primitive object to at least one window corresponding to at least one of text and line drawing operations in said window.

7. The method of claim 6, wherein said adding step comprises the step of recording the text and clipping rectangle for said text drawing operation.

8. The method of claim 5, further comprising the step of updating information concerning said graphical primitive objects to reflect modifications to said graphical primitive objects.

9. The method of claim 5, further comprising the step of updating information concerning said graphical primitive objects to reflect movement of said graphical primitive objects, said movement including at least one of the following types of movement:

copying an off-screen bitmap into a container window in said virtual window hierarchy to thereby simulate the drawing of objects corresponding to said bitmap directly in said container window;

moving a portion of said display and updating origin information concerning said moved portion; and

moving a content window within its parent window.

10. The method of claim 5, wherein said graphical primitive objects include lines and text strings and said virtual objects include text fields and composite objects.

11. The method of claim 10, wherein said composite objects include tables.

12. The method of claim 5, wherein said object grouping process thread resides in between said application under test and said graphical user interface operating system such that requests from said application under test to said graphical user interface operating system are intercepted and analyzed by said object grouping process thread before being passed on from said object grouping process thread to said graphical user interface operating system.

13. The method of claim 5, further comprising the step of adding at least one new graphical primitive object to at least one window said new graphical primitive object including a hyperlink, and said adding step further comprising the steps of positioning a mouse cursor in the vicinity of said hyperlink and extracting status window information corresponding to said hyperlink.

14. The method of claim 5, wherein said grouping step includes the use of a set of parameterized rules applied to the graphical primitive objects.

15. The method of claim 14, wherein said rules are applied in a predetermined order.

16. The method of claim 14, wherein said rules are applied iteratively such that the object grouping is continually refined by the iterative application of said rules.

17. The method of claim 14, wherein a predetermined tolerance parameter is used in the application of said rules.

18. The method of claim 5, wherein said grouping step comprises the step of creating rectangles from lines.

19. The method of claim 18, wherein said lines include horizontal and vertical lines, and said step of creating rectangles further comprises the following steps:

sorting said lines by vertical position;

selecting a first vertical line;

selecting a first horizontal line having a left end which matches a top end of said selected first vertical line; and

selecting a second vertical line having a top end which matches a right end of said selected first horizontal line, and having a bottom end which matches a bottom end of said selected first vertical line.

20. The method of claim 19, wherein said step of creating rectangles further comprises the following step:

selecting a second horizontal line which extends from the bottom end of said selected first vertical line to the bottom end of said selected second vertical line.

21. The method of claim 20, wherein said selected lines have colors, said colors being used to assign one of a raised and lowered attribute to said rectangle.

22. The method of claim 20, wherein said step of creating rectangles is repeated and a list is assembled of the rectangles constructed from individual lines.

23. The method of claim 22, wherein said list of constructed rectangles is merged with a list of drawn rectangles.

24. The method of claim 19, wherein said first vertical line, said second vertical line and said first horizontal line identify a three-sided page list control.

25. The method of claim 24, wherein a plurality of page list controls are grouped in accordance with the following steps:

the page list controls are sorted by the vertical position of a missing edge in each page list control;

the page list controls are sorted from left to right;

grouping two page list controls if at least one of the following conditions is met;

the two page list controls are bottom aligned;

the two page list controls are within a predetermined distance of each other; and

each of the page list controls is labelled with a single text object or a plurality of overlapping text objects used to create a three-dimensional effect.

26. The method of claim 5, wherein said grouping step comprises the step of merging adjacent images into a single object.

27. The method of claim 26, wherein said merging step merges two images having a same top and bottom edge, and wherein a right edge of one image matches a left edge of another image.

28. The method of claim 26, wherein said merging step merges two images having matching left and right edges, and wherein a bottom edge of one image matches a top edge of another image.

29. The method of claim 5, wherein said grouping step comprises the step of merging text objects into a single object.

30. The method of claim 29, wherein said step of merging text objects is repeated to merge a plurality of text objects into a single text object.

31. The method of claim 29, wherein said step of merging text objects further comprises the following step:

merging at least one of a list number and a bullet with a text string to produce a list object, wherein the text string meets at least one of the following conditions;

the text string is positioned within a predetermined distance to the right of said list number or bullet; and

the text string is center aligned with said list number or bullet.

32. The method of claim 31, wherein said list objects are grouped into a list if they are within a predetermined distance of each other.

33. The method of claim 32, wherein said step of grouping list objects continues until at least one of the following conditions is met:

a non-list object is encountered;

a horizontal line is encountered; and

a list object is encountered which begins to the left of the first list object in the group.

34. The method of claim 31, wherein said step of merging text objects includes the step of merging a text object ending with a colon with an adjacent text object, if at least one of the following conditions is met:

the text object is below the text object ending with the colon;

the text object is within a predetermined distance of the text object ending with a colon; and

the text object is left aligned with the text object ending with a colon.

35. The method of claim 29, wherein said step of merging text objects further comprises the following step:

merging two objects which overlap vertically.

36. The method of claim 29, wherein said step of merging text objects further comprises the following step:

merging two text objects which are stacked, wherein a bottom of one text object is in the vicinity of a top of another text object, if the two text objects satisfy at least one of the following conditions;

the text objects are left aligned;

the text objects are right aligned; and

the text objects are center aligned.

37. The method of claim 36, wherein the step of merging two text objects is prevented if at least one of the following conditions is met:

a line is positioned between said objects;

said objects end with a colon; and

only one object is in bold face type.

38. A method for recognizing graphical objects and grouping said graphical objects into a table having at least one row and at least one column, said method comprising the following steps:

identifying graphical objects as graphical object primitives;

grouping the graphical primitive objects in a window using an object grouping process thread in order to create virtual objects, said grouping step comprising the step of creating rectangles from lines;

constructing a list of cells in said window, wherein said cells comprise rectangles;

sorting said cells by their vertical position, and then left to right for cells with the same vertical position;

dividing said cells into rows based on the vertical positions of said cells, the cells on a given row having substantially similar vertical positions, said dividing step including the step of identifying a row break when the top of a cell being considered is below the bottom of a cell in the row having the highest bottom position; and

creating links for at least one cell in the table, said links including a below link which points to the next cell in the same column, and a right link which points to the cell which is the next cell in the same row.

39. The method of claim 38, further comprising the step of associating a scroll bar in said window with said table, wherein said associated scroll bar is located at an edge of the table.

40. The method of claim 38, wherein the step of dividing cells into rows includes the additional steps of locating a horizontal center line which maximizes the number of cells in a row, and adjusting the cells in a row to have substantially the same top position corresponding to the top of the leftmost cell in the row, and adjusting the cells in a row to have substantially the same bottom position corresponding to the bottom-most position of the cells in the row.

41. The method of claim 38, further comprising the step of identifying gaps corresponding to missing cells in said table by examining the below links for at least one row of said table.

42. The method of claim 41, further comprising the steps of inserting an empty cell corresponding to each said identified gap, and adjusting the links of the remaining cells to reflect the insertion of each empty cell.

43. The method of claim 42, wherein said inserting step includes the insertion of an empty cell in a gap corresponding to the left-most position of the top row of said table, said left-most gap being identified by a below link of the first cell in a row which is a target of a right link and a source of the right link and the target of the below link are top aligned.

44. The method of claim 38, further comprising the step of identifying a starting cell for said table.

45. The method of claim 38, wherein said cells are arranged in columns wherein each cell in a column is substantially left aligned with other cells in the column and located on different rows of said table.

46. The method of claim 38, further comprising the following steps:

identifying a spanning cell which is left aligned with a first cell on another row and which is right aligned with a second cell on the same other row; and

splitting said spanning cell into two or more individual cells each containing the same information as the spanning cell.

47. The method of claim 38, further comprising the step of merging a first cell on a first row which substantially overlaps a second cell on a second row.

48. The method of claim 38, wherein the step of dividing cells into rows includes the step of identifying an end of a first row of said table and advancing to a second row of said table when at least the following condition is met:

each cell in the first row has a below link to a cell which is within a predetermined distance.

49. The method of claim 48, further comprising the step of identifying the first row as a table heading if at least one of the following conditions is met:

the first row contains text which is all in bold face type and the second row does not contain all bold face type text; and

the first row contains text that is all of a first color and the second row contains text which is all of a second color and said first and second colors are different.

50. The method of claim 48, wherein an indication of a cell table is reported if the number of rows in the table exceeds a predetermined minimum number of rows.

51. The method of claim 48, wherein a column object is constructed corresponding to each column of the table, and wherein the cells of a column object are identified as children of the column object.

52. The method of claim 48, further comprising the following steps:

identifying a table heading by examining text objects that are within a predetermined distance of the top of said table and substantially centered about said table;

identifying a scroll bar associated with said table by selecting a scroll bar having a bottom which is substantially aligned with the bottom of said table, and a top which is substantially aligned with the heading or the first row of the table.

53. The method of claim 38, wherein said table is a cell table.

54. The method of claim 38, wherein said table is a text table.

55. The method of claim 54, wherein the elements in each row are substantially center aligned.