User interface automation framework classes and interfaces

• US 7,644,367 B2
• Filed: 05/16/2003
• Issued: 01/05/2010
• Est. Priority Date: 05/16/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Burns, Heather S., Friedman, Michael A., Sinclair, Robert, Wagoner, Patricia M, McKeon, Brendan, Reid, Paul J.
• Primary Examiner(s): Bashore; William L
• Assistant Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US10/439,514
• Publication Number:
  US 20040229200A1
• Time in Patent Office: 2,426 Days
• Field of Search: 715/762, 345/747
• US Class Current: 715/762
• CPC Class Codes: H04N 21/241 Operating system [OS] processes, e.g. server setup
• CPC Class Codes: H04N 21/4431 characterized by the use of Application Program Interface [API] libraries
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/8543 using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80

Abstract

A method and system providing a client with user interface information are described. The method and system implement a set of application program interfaces for providing user interface information to a client through an accessibility system. The accessibility system includes a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information. The application program interface system includes client side application program interfaces for assisting the client in obtaining user interface information. The client side application program interfaces include an automation class, a logical element class, a raw element class, control pattern classes, and an input class. The provider side application program interfaces include a provider automation class, a provider automation interface, a raw element interface, and control pattern provider interfaces.

Claims

1. A system of tools for use in an accessibility system that provides one or more clients with user interface information, the accessibility system including a client side for each of the one or more clients and a provider side, the system of tools comprising:

client side automation tools including a client automation class for seeking user interface information, the client automation class including logical element discovery tools and event registration tools, the event registration tools allowing the one or more clients to register to receive one or more user interface event notifications for user interface property changes, control structure changes, multimedia events, or a combination thereof, which thereby reduces cross-process calls; and

provider side automation tools for providing the one or more clients with user interface information, the provider side automation tools including an automation provider class having tools for providing the one or more clients with event information,

wherein, if the one or more clients register to receive the one or more user interface event notifications for a particular event and the event is associated with a user interface element visible to a user, monitoring a user interface for the particular event for which the one or more clients registered to receive the one or more user interface event notifications that matches the information request, filtering the user interface information in correspondence with the information request, creating a logical element tree based on the filtered user interface information, transmitting the logical element tree to place the logical element tree with the one or more clients requesting information, the one or more clients configured for users having disabilities, and exposing a form of the logical element tree to the end user that corresponds with the user interface being presented to the user thereby assisting the user to interact with an application.

2. The system of claim 1, further comprising client side logical element tools for representing a user interface element.

3. The system of claim 2, further comprising client side raw element tools for representing an element in a raw element tree.

4. The system of claim 1, further comprising a client side input class for providing methods for simulating input.

5. The system of claim 4, further comprising tools for simulating mouse input and keyboard input.

6. The system of claim 2, further comprising client side control pattern classes for allowing the client to interact with accessibility system control patterns.

7. The system of claim 6, wherein the client side control pattern classes include an application window pattern class for exposing information associated with a main window of an application.

8. The system of claim 6, wherein the client side control pattern classes include an expand and collapse pattern class for manipulating content that can be displayed and can be hidden.

9. The system of claim 6, wherein the client side control pattern classes include a grid item class for discovering if an item is part of a grid.

10. The system of claim 6, wherein the client side control pattern classes include a hierarchy item pattern class for representing UI elements that expose the hierarchal relationship between UI elements independent from their relationship in the logical tree.

11. The system of claim 6, wherein the client side control pattern classes include an invoke pattern class for representing objects having a single associated action.

12. The system of claim 6, wherein the client side control pattern classes include a multiple view pattern class for manipulating an element that can switch between multiple representations.

13. The system of claim 6, wherein the client side control pattern classes comprise a range value pattern class for exposing minimum and maximum values of a control.

14. The system of claim 6, wherein the client side control pattern classes comprise a scroll pattern class for manipulating an element that can alter visibility through scrolling.

15. The system of claim 6, wherein the client side control pattern classes comprise a selection pattern class for manipulating elements in a container that can be selected and unselected.

16. The system of claim 6, wherein the client side control pattern classes comprise a sort pattern class for manipulating a container element that can sort sub-elements.

17. The system of claim 6, wherein the client side control pattern classes comprise a value pattern class for representing user interface elements that express a value.

18. The system of claim 6, further comprising a window pattern class for manipulating a window on a user desktop.

19. The system of claim 1, further comprising auxiliary tools for performing additional functions.

20. The system of claim 1, further comprising a provider side automation interface for exposing additional properties.

21. The system of claim 1, further comprising a provider raw element interface for returning relative elements from a raw element tree.

22. The system of claim 1, further comprising control pattern provider interfaces for exposing behavior and information associated with defined control patterns.

23. A system of client side tools implemented in an accessibility system that provides a client with user interface information, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the client side tools comprising:

a computer processor;

a client side automation mechanism including a client automation class for seeking user interface event information from the provider side; and

a client side logical element mechanism including a logical element class for representing a user interface element in the logical tree, the logical tree comprising a filtered view of an underlying structural hierarchy having only one or more user interface elements that are of interest to a client and visible to a user, wherein each of the one or more user interface elements represents a control, an item in a control, or a grouping structure, the logical tree being utilized to identify information related to an event, a state of a system, a location of an object, a control, or a combination thereof, the logical tree being created by locating native user interface elements of underlying technologies that include elements that exist for structure and implementation to create one or more native trees, combining the native elements to form a raw tree that contains elements representing an implementation structure of an underlying framework and that indicates an unprocessed logical tree, and creating the logical tree by eliminating the native elements in the raw tree regarding object information and properties which are determined to be uninteresting to the client.

24. The system of claim 23, wherein the client automation class comprises tools for adding an automation event handler and tools for removing the automation event handler.

25. The system of claim 23, wherein the client automation class comprises tools for adding an automation property changed event handler and tools for removing the automation property changed event handler.

26. The system of claim 23, wherein the client automation class comprises tools for adding a focus changed event handler and tools for removing a focus changed event handler.

27. The system of claim 23, wherein the client automation class comprises tools for adding a top level window event handler and tools for removing a top level window event handler.

28. The system of claim 23, wherein the client automation class comprises tools for adding a logical structure changed event handler and tools for removing the logical structure changed event handler.

29. The system of claim 23, wherein the client automation class comprises tools for finding a logical element and tools for finding a raw element.

30. The system of claim 23, further comprising a client side raw element class for representing an element in a raw element tree.

31. The system of claim 23, wherein the logical element class further comprising tools for obtaining all patterns supported by an element.

32. The system of claim 23, further comprising an input class for providing methods for simulating input.

33. The system of claim 32, wherein the input comprises mouse input and keyboard input.

34. A system of provider side tools implemented in an accessibility system that provides a client with user interface information, the accessibility system including a mechanism for transferring user interface information from the provider side to the client and a logical tree for selectively revealing user interface information, the provider side tools comprising:

a computer processor;

a provider side automation class including tools for providing user interface event notifications to the client, the user interface event notifications indicating user interface property changes, control structure changes, multimedia events, or a combination thereof wherein, if the client requests information for a particular event and the event is associated with a user interface element visible to a user, monitoring a user interface for the particular event that matches the information request, filtering the user interface information in correspondence with the information request, creating a logical element tree based on the filtered user interface information, and transmitting the logical element tree to place the logical element tree with the client requesting information, the client configured for users having disabilities;

a provider side automation interface for exposing user interface properties;

a raw element interface for returning information related to a specific relative element; and

a raw element context interface for managing events and functionality not related to a specific element.

35. The system of claim 34, wherein the provider side automation class includes a tool for raising an automation event.

36. The system of claim 34, wherein the provider side automation class includes a tool for raising a property changed event.

37. The system of claim 34, wherein the provider side automation class includes a tool for raising a focus changed event.

38. The system of claim 34, wherein the provider side automation class includes a tool for raising a logical structure changed event.

39. The system of claim 34, further comprising control pattern provider interfaces.

40. The system of claim 39, wherein the control pattern provider interfaces include an application window provider interface for exposing behavior and information associated with a top level application window.

41. The system of claim 39, wherein the control pattern provider interfaces include an expand and collapse interface for hiding and displaying content.

42. The system of claim 39, wherein the control pattern provider interfaces include a grid provider interface for exposing basic grid functionality.

43. The system of claim 39, wherein the control pattern provider interfaces include a hierarchy item provider interface for allowing the client to traverse a hierarchical relationship between user interface elements.

44. The system of claim 39, wherein the control pattern provider interfaces include an invoke provider interface for use by objects that perform a single action.

45. The system of claim 39, wherein the control pattern provider interfaces include a multiple view provider interface for exposing a capability of an object to switch between multiple representations.

46. The system of claim 39, wherein the control pattern provider interfaces include a range value provider interface for exposing a set of properties allowing an ability to manage a finite range of values.

47. The system of claim 39, wherein the control pattern provider interfaces include a scroll provider interface for exposing an ability to change a visible region.

48. The system of claim 39, wherein the control pattern provider interfaces include a selection by identification provider for exposing an element that offers a choice between items.

49. The system of claim 39, wherein the control pattern provider interfaces include a selection provider interface for representing a container that manages selection.

50. The system of claim 39, wherein the control pattern provider interfaces include a window provider interface for exposing an ability to change size and position.

51. An application program interface system for providing user interface information to a client through an accessibility system, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the application program interface system comprising:

a computer processor;

client side application program interfaces for assisting the client in obtaining user interface information, wherein the client side application program interfaces include an automation class, a raw element class;

control pattern classes, an input class, and a logical element class for representing a user interface element in the logical tree, the logical tree comprising a filtered view of an underlying structural hierarchy having only one or more user interface elements that are of interest to a client and visible to a user, wherein each of the one or more user interface elements represents a control, an item in a control, or a grouping structure, the logical tree being utilized to identify information related to an event, a state of a system, a location of an object, a control, or a combination thereof, the logical tree being created by locating native user interface elements of underlying technologies that include elements that exist for structure and implementation to create one or more native trees, combining the native elements to form a raw tree that contains elements representing an implementation structure of an underlying framework and that indicates an unprocessed logical tree, and creating the logical tree by eliminating the native elements in the raw tree regarding object information and properties which are determined to be uninteresting to the client; and

provider side application program interfaces for responding to client requests, the provider side application program interfaces comprising a provider automation class, a provider automation interface, a raw element interface, and control pattern provider interfaces.

52. The system of claim 51, further comprising auxiliary classes used by the provider side and the client side.

53. A computer-implemented method for providing user interface information to a client through an accessibility system, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the method comprising

providing client side application program interfaces for assisting the client in obtaining user interface information, wherein the client side application program interfaces include an automation class, a raw element class, control pattern classes, an input class, and a logical element class for representing a user interface element in the logical tree, the logical tree comprising a filtered view of an underlying structural hierarchy having only one or more user interface elements that are of interest to a client and visible to a user, wherein each of the one or more user interface elements represents a control, an item in a control, or a grouping structure, the logical tree being utilized to identify information related to an event, a state of a system, a location of an object, a control, or a combination thereof, the logical tree being created by locating native user interface elements of underlying technologies that include elements that exist for structure and implementation to create one or more native trees, combining the native elements to form a raw tree that contains elements representing an implementation structure of an underlying framework and that indicates an unprocessed logical tree, and creating the logical tree by eliminating the native elements in the raw tree regarding object information and properties which are determined to be uninteresting to the client;

and

supplying provider side application program interfaces for responding to client requests, the provider side application program interfaces comprising a provider automation class, a provider automation interface, a raw element interface, and control pattern provider interfaces.

54. The method of claim 53, further comprising providing auxiliary classes for use by the provider side and the client side.