User interface automation framework classes and interfaces

• US 20040229200A1
• Filed: 05/16/2003
• Published: 11/18/2004
• Est. Priority Date: 05/16/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Burns, Heather S., McKeon, Brendan, Friedman, Michael A., Sinclair, Robert, Reid, Paul J., Wagoner, Patricia M.
• Granted Patent: US 7,644,367 B2
• Time in Patent Office: Days
• US Class Current: 434/350
• CPC Class Codes: H04N 21/241 Operating system [OS] processes, e.g. server setup
• CPC Class Codes: H04N 21/4431 characterized by the use of Application Program Interface [API] libraries
• CPC Class Codes: H04N 21/8173 End-user applications, e.g. Web browser, game
• CPC Class Codes: H04N 21/8543 using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML] information retrieval of semistructured data, the underlying structure being taken into account, e.g. mark-up language structure data G06F16/80

Abstract

A method and system providing a client with user interface information are described. The method and system implement a set of application program interfaces for providing user interface information to a client through an accessibility system. The accessibility system includes a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information. The application program interface system includes client side application program interfaces for assisting the client in obtaining user interface information. The client side application program interfaces include an automation class, a logical element class, a raw element class, control pattern classes, and an input class. The provider side application program interfaces include a provider automation class, a provider automation interface, a raw element interface, and control pattern provider interfaces.

Claims

1. A system of tools for use in an accessibility system that provides a client with user interface information, the accessibility system including a client side and a provider side, the system of tools comprising:

client side automation tools including a client automation class for seeking user interface information, the client automation class including event registration tools and logical element discovery tools; and

provider side automation tools for providing the client with user interface information, the provider side automation tools including an automation provider class having tools for providing the client with event information.

2. The system of claim 1, further comprising client side logical element tools for representing a user interface element.

3. The system of claim 2, further comprising client side raw element tools for representing an element in a raw element tree.

4. The system of claim 1, further comprising a client side input class for providing methods for simulating input.

5. The system of claim 4, further comprising tools for simulating mouse input and keyboard input.

6. The system of claim 2, further comprising client side control pattern classes for allowing the client to interact with accessibility system control patterns.

7. The system of claim 6, wherein the client side control pattern classes include an application window pattern class for exposing information associated with a main window of an application

8. The system of claim 6, wherein the client side control pattern classes include an expand and collapse pattern class for manipulating content that can be displayed and can be hidden.

9. The system of claim 6, wherein the client side control pattern classes include a grid item class for discovering if an item is part of a grid.

10. The system of claim 6, wherein the client side control pattern classes include a hierarchy item pattern class for representing UI elements that expose the hierarchal relationship between UI elements independent from their relationship in the logical tree.

11. The system of claim 6, wherein the client side control pattern classes include an invoke pattern class for representing objects having a single associated action.

12. The system of claim 6, wherein the client side control pattern classes include a multiple view pattern class for manipulating an element that can switch between multiple representations.

13. The system of claim 6, wherein the client side control pattern classes comprise a range value pattern class for exposing minimum and maximum values of a control.

14. The system of claim 6, wherein the client side control pattern classes comprise a scroll pattern class for manipulating an element that can alter visibility through scrolling.

15. The system of claim 6, wherein the client side control pattern classes comprise a selection pattern class for manipulating elements in a container that can be selected and unselected.

16. The system of claim 6, wherein the client side control pattern classes comprise a sort pattern class for manipulating a container element that can sort sub-elements.

17. The system of claim 6, wherein the client side control pattern classes comprise a value pattern class for representing user interface elements that express a value.

18. The system of claim 6, further comprising a window pattern class for manipulating a window on a user desktop.

19. The system of claim 1, further comprising auxiliary tools for performing additional functions.

20. The system of claim 1, further comprising a provider side automation interface for exposing additional properties.

21. The system of claim 1, further comprising a provider raw element interface for returning relative elements from a raw element tree.

22. The system of claim 1, further comprising control pattern provider interfaces for exposing behavior and information associated with defined control patterns.

23. A system of client side tools implemented in an accessibility system that provides a client with user interface information, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the client side tools comprising:

a client side automation mechanism including a client automation class for seeking user interface event information from the provider side; and

a client side logical element mechanism including a logical element class for representing a user interface element in the logical tree.

24. The system of claim 23, wherein the client automation class comprises tools for adding an automation event handler and tools for removing the automation event handler.

25. The system of claim 23, wherein the client automation class comprises tools for adding an automation property changed event handler and tools for removing the automation property changed event handler.

26. The system of claim 23, wherein the client automation class comprises tools for adding a focus changed event handler and tools for removing a focus changed event handler.

27. The system of claim 23, wherein the client automation class comprises tools for adding a top level window event handler and tools for removing a top level window event handler.

28. The system of claim 23, wherein the client automation class comprises tools for adding a logical structure changed event handler and tools for removing the logical structure changed event handler.

29. The system of claim 23, wherein the client automation class comprises tools for finding a logical element and tools for finding a raw element.

30. The system of claim 23, further comprising a client side raw element class for representing an element in a raw element tree.

31. The system of claim 23, wherein the logical element class further comprising tools for obtaining all patterns supported by an element.

32. The system of claim 23, further comprising an input class for providing methods for simulating input.

33. The system of claim 32, wherein the input comprises mouse input and keyboard input.

34. A system of provider side tools implemented in an accessibility system that provides a client with user interface information, the accessibility system including a mechanism for transferring user interface information from the provider side to the client and a logical tree for selectively revealing user interface information, the provider side tools comprising:

a provider side automation class including tools for providing event notifications to the client;

a provider side automation interface for exposing user interface properties;

a raw element interface for returning information related to a specific relative element; and

a raw element context interface for managing events and functionality not related to a specific element.

35. The system of claim 34, wherein the provider side automation class includes a tool for raising an automation event.

36. The system of claim 34, wherein the provider side automation class includes a tool for raising a property changed event.

37. The system of claim 34, wherein the provider side automation class includes a tool for raising a focus changed event.

38. The system of claim 34, wherein the provider side automation class includes a tool for raising a logical structure changed event.

39. The system of claim 34, further comprising control pattern provider interfaces.

40. The system of claim 39, wherein the control pattern provider interfaces include an application window provider interface for exposing behavior and information associated with a top level application window.

41. The system of claim 39, wherein the control pattern provider interfaces include an expand and collapse interface for hiding and displaying content.

42. The system of claim 39, wherein the control pattern provider interfaces include a grid provider interface for exposing basic grid functionality.

43. The system of claim 39, wherein the control pattern provider interfaces include a hierarchy item provider interface for allowing the client to traverse a hierarchical relationship between user interface elements.

44. The system of claim 39, wherein the control pattern provider interfaces include an invoke provider interface for use by objects that perform a single action.

45. The system of claim 39, wherein the control pattern provider interfaces include a multiple view provider interface for exposing a capability of an object to switch between multiple representations.

46. The system of claim 39, wherein the control pattern provider interfaces include a range value provider interface for exposing a set of properties allowing an ability to manage a finite range of values.

47. The system of claim 39, wherein the control pattern provider interfaces include a scroll provider interface for exposing an ability to change a visible region.

48. The system of claim 39, wherein the control pattern provider interfaces include a selection by identification provider for exposing an element that offers a choice between items.

49. The system of claim 39, wherein the control pattern provider interfaces include a selection provider interface for representing a container that manages selection.

50. The system of claim 39, wherein the control pattern provider interfaces include a window provider interface for exposing an ability to change size and position.

51. An application program interface system for providing user interface information to a client through an accessibility system, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the application program interface system comprising:

client side application program interfaces for assisting the client in obtaining user interface information, wherein the client side application program interfaces include an automation class, a logical element class, a raw element class, control pattern classes, and an input class; and

provider side application program interfaces for responding to client requests, the provider side application program interfaces comprising a provider automation class, a provider automation interface, a raw element interface, and control pattern provider interfaces.

52. The system of claim 51, further comprising auxiliary classes used by the provider side and the client side.

53. A computer-implemented method for providing user interface information to a client through an accessibility system, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the method comprising

providing client side application program interfaces for assisting the client in obtaining user interface information, wherein the client side application program interfaces include an automation class, a logical element class, a raw element class, control pattern classes, and an input class;

and

supplying provider side application program interfaces for responding to client requests, the provider side application program interfaces comprising a provider automation class, a provider automation interface, a raw element interface, and control pattern provider interfaces.

54. The method of claim 53, further comprising providing auxiliary classes for use by the provider side and the client side.

55. A computer-implemented method for providing user interface information to a client through an accessibility system, the accessibility system including a mechanism for transferring user interface information from a provider side to the client side and a logical tree for selectively revealing user interface information, the method comprising:

requesting user interface information using a selected event handler from a client automation class; and

invoking a provider automation class to provide user interface event notifications using a corresponding raise event method.

56. The method of claim 55, further comprising using logical element tools to obtain user interface element information.

57. The method of claim 55, further comprising using client side raw element tools for obtaining raw element information regarding an element in a raw element tree.

58. The method of claim 55, further comprising utilizing a client side input class for providing methods for simulating input.

59. The method of claim 58, further comprising simulating mouse input and keyboard input.

60. The method of claim 55, further comprising using client side control pattern classes for allowing the client to interact with accessibility system control patterns.

61. The method of claim 55, further comprising implementing auxiliary tools for performing additional functions.

62. The method of claim 55, further comprising utilizing a provider side automation interface for exposing additional properties.

63. The method of claim 55, comprising implementing a provider raw element interface for returning relative elements from a raw element tree.

64. The method of claim 55, further comprising implementing control pattern provider interfaces for exposing behavior and information associated with defined control patterns.