SYSTEMS AND METHODS FOR IMPLEMENTING PIXEL-BASED REVERSE ENGINEERING OF INTERFACE STRUCTURE

• US 20110126158A1
• Filed: 11/23/2010
• Published: 05/26/2011
• Est. Priority Date: 11/23/2009
• Status: Active Grant

• Current Assignee: University of Washington
• Original Assignee: University of Washington
• Inventors: Leventhal, Daniel, Dixon, Morgan, Fogarty, James
• Granted Patent: US 9,182,981 B2
• Time in Patent Office: Days
• US Class Current: 715/835
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 8/75 Structural analysis for program understanding
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images

Abstract

Systems and methods for implementing real-time pixel-based reverse engineering of interface structures are disclosed herein. The system uses the pixels representing widgets and content of a user interface to analyze, interpret, and/or enhance the interface without requiring cooperation from the interface. The system captures pixel data and reverse engineers the composition and structure of the interface from the captured pixel data. Using the reverse engineered interface, generated based on the captured pixel data, the facility can identify and record user interactions with the interface, modify the display of the interface to meet user requirements, preferences, or customization options, or enhance the interface to provide functionality not otherwise available, etc. The system can perform these processes across a variety of interfaces regardless of their underlying implementation and without any modifications to the interface.

Claims

1. A method, performed by a computer having a memory and a processor, for interpreting the contents of a user interface without requiring cooperation from the user interface, the method comprising:

capturing pixel data of a source window associated with the user interface;

with a processor, identifying areas of the captured pixel data that at least substantially match features associated with at least one widget prototype of a prototype library; and

with a processor, identifying a user interface widget within the captured pixel data at least in part by determining whether a spatial relationship between at least two of the identified areas is consistent with at least one widget prototype of the prototype library.

2. The method of claim 1 wherein identifying a user interface widget within the captured pixel data further comprises:

testing the color and arrangement of selected pixels of the captured pixel data other than pixels of the at least two identified areas to determine whether the selected pixels are consistent with at least one widget prototype of the prototype library.

3. The method of claim 2 wherein at least one widget prototype defines an arrangement of features associated with widgets of the user interface.

4. The method of claim 3 wherein at least one widget prototype defines an arrangement of regions relative to features associated with widgets of the user interface, individual regions specifying a repeating pixel, a repeating pattern of pixels, or a gradient.

5. The method of claim 1, further comprising:

altering a portion of the captured pixel data based at least in part on the identified user interface widget; and

displaying the altered pixel data to enable modification of apparent contents of the user interface.

6. The method of claim 5, further comprising:

updating contents of the altered pixel data based at least in part on a change to the user interface of the source window.

7. The method of claim 5 wherein the altered pixel data is not displayed at the source window.

8. The method of claim 5, further comprising:

identifying text associated with the identified user interface widget, the identified text being in a first language; and

receiving a translation of the identified text in a second language,

wherein altering a portion of the captured pixel data includes replacing pixels of the pixel data corresponding to the identified text in the first language with pixels corresponding to the translation of the identified text in the second language.

9. The method of claim 5 wherein modifying the pixel data includes representing a new widget within the pixel data that is not represented in the captured pixel data.

10. The method of claim 1, further comprising:

receiving user input;

mapping the received user input to a widget of the user interface of the source window based at least in part on the captured pixel data and the identified user interface widget; and

manipulating the user interface of the source window based on the mapped user input to enable modification of the apparent behavior of the user interface.

11. The method of claim 10 wherein the user input is not received at the source window.

12. The method of claim 1, further comprising:

in response to determining that the spatial relationship between the at least two of the identified areas is consistent with a first widget prototype and a second widget prototype, selecting a widget prototype corresponding to the user interface widget at least in part by,

determining the number of pixels of the user interface widget that do not match pixels of a content region of the first widget prototype, and

determining the number of pixels of the user interface widget that do not match pixels of a content region of the second widget prototype.

13. The method of claim 1, further comprising:

generating input to manipulate the identified user interface widget.

14. The method of claim 13 wherein the generated input is generated in response to user interaction with an indication of the identified user interface widget or data associated with the identified user interface widget.

15. A computer-readable storage medium containing instructions that, when executed by a computer, perform steps comprising:

capturing pixel data of an interface;

identifying features within the captured pixel data, each feature having an arrangement of pixels; and

identifying at least one interface widget within the captured pixel data at least in part by determining whether a spatial relationship between at least two of the identified features is consistent with at least one widget prototype of the prototype library, each widget prototype specifying at least one arrangement of features.

16. The computer-readable storage medium of claim 15, further comprising:

generating at least one widget prototype at least in part by,

displaying an image of at least part of the interface,

receiving, from a user, an indication of a location corresponding to a widget within the displayed image,

identifying, based at least in part on the location, an area of the displayed image corresponding to the widget, and

storing at least a portion of the pixels of the identified area as at least part of at least one widget prototype.

17. The computer-readable storage medium of claim 16 wherein identifying the area of the displayed image corresponding to the widget is based at least in part on a gradient threshold.

18. The computer-readable storage medium of claim 15, further comprising:

determining at least one widget prototype at least in part by;

receiving a first candidate for the at least one widget prototype;

receiving a second candidate for the at least one widget prototype;

determining the number of pixels associated with the first candidate;

determining the number of pixels associated with the second candidate; and

selecting as the at least one widget prototype the first candidate or the second candidate based at least in part on which candidate has the fewest number of associated pixels.

19. The computer-readable storage medium of claim 15, further comprising:

constructing a portion of an interface widget hierarchy at least in part by,

storing an indication of a first interface widget as a first node of the interface widget hierarchy,

identifying, within a portion of the captured pixel data corresponding to the first interface widget, a plurality of second user interface widgets at least in part by comparing pixels of the captured pixel data to widget prototypes selected from the prototype library, and

for each identified second user interface widget, storing an indication of the identified second user interface widget as a child node of the first node.

20. The computer-readable storage medium of claim 15, further comprising:

receiving image data including a user interface; and

creating a user interface widget hierarchy for the user interface of the received image data at least in part by,

creating a root node for the user interface widget hierarchy,

analyzing the received image data to identify each widget of the user interface,

creating a node for each identified widget of the user interface, and for each identified widget,

in response to determining that the identified widget is contained by at least one widget, adding the node of the identified widget as a child of the node of one widget containing the identified widget, and

in response to determining that the identified widget is not contained by a widget, adding the node of the identified widget as a child of the root node.

21. The computer-readable storage medium of claim 15 wherein the identified at least one user interface widget is one of a button widget, a checkbox widget, a combobox widget, a label widget, a list widget, a menu widget, a progressbar widget, a scrollbar widget, a slider widget, a spinner widget, a split pane widget, a tabbed pane widget, a textfield widget, a tooltip widget, or a tree widget.

22. The computer-readable storage medium of claim 15, the steps further comprising:

identifying unpredictable content within the at least one interface widget at least in part by comparing a content region associated with the at least one widget prototype with pixels of the at least one interface widget.

23. The computer-readable storage medium of claim 22 wherein the unpredictable content is text.

24. The computer-readable storage medium of claim 15 wherein the interface has an associated application programming interface.

25. The computer-readable storage medium of claim 24, the steps further comprising:

retrieving a hierarchical description of widgets of at least a portion of the interface via the associated application programming interface,

wherein identifying pixel data of the interface includes identifying at least one widget not represented in the retrieved hierarchical description.

26. The computer-readable storage medium of claim 25, further comprising:

providing an enhanced representation of at least one widget represented in the retrieved hierarchical description or at least one widget not represented in the retrieved hierarchical description.

27. A computing system comprising:

a component configured to identify interface widget features within captured pixel data corresponding to a first interface;

a component configured to identify a plurality of widget prototypes of a prototype library based on the identified interface widget features, each widget prototype specifying at least two features and having at least one associated constraint specifying a spatial relationship associated with at least one feature of the widget prototype; and

a component configured to, in response to detecting a widget within the captured pixel data by comparing at least one of the identified widget prototypes to the captured pixel data, storing an indication of the detected widget.

28. The computing system of claim 27 wherein the prototype library includes at least one widget prototype associated with each of a plurality of types of user interface widgets in one or more configurations.

29. The computing system of claim 28 wherein the types of user interface widgets include at least one of a button widget, a checkbox widget, a combobox widget, a label widget, a list widget, a menu widget, a progressbar widget, a scrollbar widget, a slider widget, a spinner widget, a split pane widget, a tabbed pane widget, a textfield widget, a tooltip widget, or a tree widget.

30. The computing system of claim 27 wherein the stored indication of the detected widget includes a widget type, a widget style, pixel data corresponding to the detected widget, and a location relative to the captured pixel data.

31. The computing system of claim 27 further comprising:

a component configured to identify unpredictable content within the detected widget at least in part by comparing pixels of the detected widget with a background region associated with the detected widget.

32. The computing system of claim 31 wherein the unpredictable content is an image.

33. The computing system of claim 27, further comprising:

a component configured to create an adapted interface based at least in part on the first interface and preferences of a user.

34. The computing system of claim 33 wherein the adapted interface is further based on the device on which the adapted interface is to be displayed.

35. The computing system of claim 33 wherein the adapted interface is adapted to provide at least one of improved accessibility, translation, localization, or security restrictions.

36. The computing system of claim 33 wherein the adapted interface is adapted to remove undesired functionality.

37. The computing system of claim 27 wherein the pixel data is captured from an archive of images.

38. The computing system of claim 37 wherein the archive is video data of the interface.

39. The computing system of claim 37, further comprising:

a component configured to associate metadata with at least one image of the archive.

40. The computing system of claim 37, further comprising:

a component configured to modify contents of at least one image.

41. The computing system of claim 40 wherein the component configured to modify contents of the at least one image is configured to crop the at least one image.