Methods, systems, and computer readable media for automatically generating customizable user interfaces using programming patterns

• US 8,752,011 B2
• Filed: 03/20/2008
• Issued: 06/10/2014
• Est. Priority Date: 03/20/2007
• Status: Active Grant

• Current Assignee: University of South Carolina
• Original Assignee: University of North Carolina At Chapel Hill (University of North Carolina System)
• Inventors: Dewan, Prasun, Omojokun, Olufisayo
• Primary Examiner(s): Bullock, Jr., Lewis A
• Assistant Examiner(s): GOORAY, MARK A
• Application Number:
  US12/532,327
• Publication Number:
  US 20100138778A1
• Time in Patent Office: 2,273 Days
• Field of Search: None
• US Class Current: 717/113
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Methods, systems and computer readable media for automatically generating customizable user interfaces using programming patterns are disclosed. According to one method, application objects of an application computer program are analyzed to identify programming patterns. Each programming pattern is a relationship among signatures of application methods in one of the application objects. The signatures of the application methods include public interfaces of the application methods. A user interface is automatically generated for the computer program. Automatically generating a user interface includes providing for at least one of a user and a programmer of the computer program to customize mappings between the application objects and user interface elements, where the user interface elements include at least one of user interface widgets and speech grammar rules.

Claims

1. A method for automatically generating a user interface for a computer program using programming patterns, the method comprising:

analyzing application objects of an application computer program to identify programming patterns, wherein each programming pattern is a relationship among signatures of application methods in one of the application objects, and wherein the signatures of the application methods comprise public interfaces of the application methods;

automatically generating a user interface for the computer program, wherein automatically generating a user interface includes providing for at least one of a user and a programmer of the computer program to customize mappings between the application objects and user interface elements, wherein the user interface elements include at least one of user interface widgets and speech grammar rules, wherein the identified programming patterns include at least one of an undo pattern, a structure pattern, a graphical pattern, a validation pattern, and a precondition pattern, and wherein the undo pattern identifies, for each application method a pattern-based executed-command-object that can undo/redo the method, wherein the pattern-based executed-command-object comprises an executed-command object that uses at least one of;

(a) relationships among the signatures of the application methods in and (b) antonym dictionaries to implement the undo and redo operations, and wherein the executed-command object is an object that provides operations to undo and redo each application method.

2. The method of claim 1 wherein identifying the programming patterns includes invoking pattern objects that map the signatures of the methods to properties.

3. The method of claim 2 wherein providing for one of a user and a programmer to customize mappings between the application objects and the user interface elements includes providing a programmer interface for the programmer to define and select the pattern objects to be invoked for given application objects.

4. The method of claim 3 wherein providing the programmer interface includes providing customization mechanisms that allow the programmer to determine:

(a) logical components and application methods of each application object are mapped to, (b) the set of user interface elements to which each mapped component and method is mapped, (c) the 2-d layout used to compose children user interface elements in containers, (d) a linear order in which speech choices are prompted in the system-initiative dialogues, (e) labels displayed in the user interface widgets and prompts used in speech dialogues, (f) whether an application method is undoable, and (g) how application methods are mapped to hierarchies of control widgets, including nested pull-down menus.

5. The method of claim 4 wherein providing customization mechanisms includes providing pattern customization mechanisms that allow programmers to specify how relationships among the method signatures should be used to identify (a) a precondition method, a validation method, and a pattern-based executed-command object, and (b) read, write, add and remove methods of a logical component.

6. The method of claim 2 wherein providing for one of a user and a programmer to customize mappings between the application objects and the user interface elements includes providing a tool as part of the user interface for a user to select the pattern objects to be involved for given application objects.

7. The method of claim 1 wherein the precondition pattern identifies a precondition method of one of the application methods, wherein the precondition method is a Boolean parameterless method that determines if the associated application method is currently enabled.

8. The method of claim 1 wherein the validation pattern identifies a validation method of one of the application methods, wherein the validation method is a Boolean method that takes the same parameters as the associated application method and determines if the application method can be safely invoked with a given set of parameters.

9. The method of claim 1 wherein the structure pattern identifies the logical components of an object, and the read, write, add and remove methods of each identified logical component.

10. The method of claim 1 wherein the graphical pattern identifies a kind of shape to be displayed by one of the application objects and a set of application methods of the application object for reading and changing attributes of the shape.

11. The method of claim 1 wherein automatically generating a user interface comprises automatically generating at least one of a graphical user interface and a speech user interface.

12. The method of claim 1 wherein automatically generating a user interface includes automatically generating one of a binder tool that binds an arbitrary set of application methods and logical components of the program to corresponding mappable user interface elements, an undo tool for implementing the undo and the redo operation, and a help tool for explaining the computer program.

13. The method of claim 12 wherein the mappable user interface elements include (a) standard state widgets provided by a user interface toolkit, the standard stated widgets including:

text-fields, formatted-text-fields, text-areas, check-boxes, radio buttons, sliders, progress bars, spinners, combo-box menus, list widgets, table widgets, and tree widgets, (b) standard control widgets including;

buttons, pop-up menus, and pull-down menus, (c) a drawing state widget including shapes of at least two dimensions representing graphical application objects, and (d) compositions of other widgets displayed in the standard containers;

such as scroll-panes, tabbed panes, and desktop-panes.

14. The method of claim 12 wherein automatically generating a help tool includes creating help information for an application, including (a) for each method or logical component mapped to a widget, showing an explanation annotation associated with the method, (b) providing commands in the user interface to display documents specified by URL annotations associated with the class and methods of the application, and (c) creating a keyword index that shows a list of keywords, and for each keyword, a list of application classes associated with the keyword, and for each class, providing the user with facilities to display its annotations and create a user interface for an instance of the class.

15. The method of claim 14 wherein the keywords include programmer-defined keywords and predefined keywords.

16. The method of claim 15 wherein the programmer-defined keywords include keywords mentioned in keyword annotations.

17. The method of claim 15 wherein the predefined keywords identify different kinds of programming patterns, widgets, and customization mechanisms.

18. The method of claim 15 wherein:

a programmer-defined keyword is associated with all classes that are mentioned in keyword annotations associated with the classes;

a predefined programming pattern keyword is associated with applications that implement the pattern;

a widget type keyword is associated with applications whose user-interfaces include the widget; and

a customization mechanism is associated with classes that use the mechanism.

19. The method of claim 12 wherein the binder tool maps application methods and logical components to mappable user interface elements, updates the mappings in response to changes to application state, and updates application state in response to user-changes to editable widgets and spoken commands.

20. The method of claim 19 wherein updating the mappings in response to changes in application state includes:

(a) when a precondition method of a mapped method returns false, removing or disabling the corresponding user interface element, (b) when a precondition method associated with a read method of a mapped logical component returns false, removing or disabling the user interface element to which it is mapped, and (c) updating an existing user interface element or adding a new user interface element when a logical component is added/removed from an object.

21. The method of claim 19 wherein updating a logical component in response to user changes to editable widgets and spoken commands includes (a) calling a validation method of a write method of the logical component to determine if the user input is valid, and (b) calling the write method if the validation method returns true.

22. The method of claim 12 wherein mapping an application method to a widget includes (a) selecting and creating a control widget so that activating the control widget results in invocation of the application method mapped to the control widget, (b) providing a dialogue box for collecting parameters of the application method, and (c) generating a user-interface to display an object returned by the application method.

23. The method of claim 12 wherein implementing an undo/redo operation includes one of (a) intercepting invocation of each application method bound to a control widget and each write method of a logical component bound to a state widget, (b) after execution of each of the write and application methods, waiting to receive an executed-command object from the write and application method as a notification or creating a pattern-based executed-command object if such a notification is not received, (c) associating the received/created executed-command object with the invoked method, (d) when the user requests the undo or redo of a method, invoking the undo or redo operation on the executed-command object associated with the invoked method.

24. The method of claim 1 wherein automatically generating the user interface includes mapping a logical component to a widget, wherein mapping a logical component to a widget includes selecting and creating a corresponding widget that can display the logical component and enabling the widget for editing if a write method of the logical component exists.

25. The method of claim 24 wherein mapping a logical component to a widget includes one of:

if a logical component P is atomic, mapping P to a standard state widget, and, if P is non-atomic, creating a container including of children widgets to which logical components and methods of P are recursively mapped.

26. The method of claim 1 wherein automatically generating a user interface includes mapping one of the application methods to a grammar rule, wherein mapping the application method to a grammar rule involves defining a system-initiative and user-initiative sub-dialogue that allows the user to identify the name of the application method and its parameters.

27. The method of claim 1 wherein automatically generating a user interface includes mapping a logical component, P, to a speech grammar rule, wherein mapping a logical component, P, to a speech grammar rule includes defining a system-initiative and user-initiative sub-dialogue that allows the user to query or change (a) P, if the logical component is atomic, and (b) the logical components of P, if the logical component is non-atomic.

28. The method of claim 27 wherein a system-initiative dialogue prompts users for the alternative choices, in order of the values of their position attributes.

29. The method of claim 28 wherein a user-initiative dialogue allows the user to give the choice without any prompting.

30. The method of claim 1 wherein analyzing the computer program to identify an undo pattern includes identifying a signature for a method that sets a value for a variable and wherein generating the user interface tool for implementing the undo operation includes, prior to execution of the method that sets the value for the variable, executing a method that stores a current value of the variable and, after execution of the method, that sets the value for the variable, providing access, via the user interface tool, to the method that sets the value for the variable where the stored value for the variable is passed as the parameter used by the method to set the value for the variable.

31. The method of claim 1 wherein analyzing the computer program to identify an undo pattern includes analyzing method names to identify names that are antonyms of each other and wherein generating the user interface tool for implementing the undo operation includes, after execution of a method M with an antonym A, providing the ability, via the user interface tool, for the user to execute the antonym A.

32. The method of claim 31 wherein analyzing method names to identify methods that are antonyms of each other includes identifying prefixes or words that indicate that the names are antonyms.

33. The method of claim 1 wherein analyzing the computer program to identify an undo pattern includes identifying a method that adds an item to a collection of items and a method that removes and item from the collection of items and wherein automatically generating the tool for implementing the undo function includes after execution of the method that adds the item to the collection of items, providing access, via the tool, to the method that removes the item from the collection of items.

34. A method for automatically generating a user interface for a computer program, the method comprising:

analyzing an application computer program to identify programming patterns, each programming pattern being a relationship among methods in an application object in the computer program that is identifiable based on signatures of the methods, the identified programming patterns including predefined programming patterns provided by a system and programmer defined patterns specified by application programmers, the signatures of the methods comprise public interfaces of the methods;

determining, from the programming patterns, properties about the application object usable for generating a user interface; and

automatically generating, based on the properties a user interface for the computer program, wherein the identified programming patterns include at least one of an undo pattern, a structure pattern, a graphical pattern, a validation pattern, and a precondition pattern, and wherein the undo pattern identifies, for each application method a pattern-based executed-command-object that can undo/redo the method, wherein the pattern-based executed-command-object comprises an executed-command object that uses at least one of;

(a) relationships among the signatures of the application methods in and (b) antonym dictionaries to implement the undo and redo operations, and wherein the executed-command object is an object that provides operations to undo and redo each application method.

35. A system for automatically generating a user interface for a computer program using programming patterns, the system comprising:

a processor;

a user interface generator implemented by the processor for analyzing application objects of an application computer program to identify programming patterns, wherein each programming pattern is a relationship among signatures of application methods in one of the application objects, and wherein the signatures of the application methods comprise public interfaces of the application methods and for automatically generating a user interface for the computer program; and

the user interface generator including a customizer for providing for at least one of a user and a programmer of the computer program to customize mappings between the application objects and user interface elements, wherein the user interface elements include at least one of user interface widgets and speech grammar rules, wherein the identified programming patterns include at least one of an undo pattern, a structure pattern, a graphical pattern, a validation pattern, and a precondition pattern, and wherein the undo pattern identifies, for each application method a pattern-based executed-command-object that can undo/redo the method, wherein the pattern-based executed-command-object comprises an executed-command object that uses at least one of;

(a) relationships among the signatures of the application methods in and (b) antonym dictionaries to implement the undo and redo operations, and wherein the executed-command object is an object that provides operations to undo and redo each application method.

36. A non-transitory computer readable medium having stored thereon a computer program comprising computer-executable instructions that when executed by the processor of a computer perform steps comprising:

analyzing application objects of an application computer program to identify programming patterns, wherein each programming pattern is a relationship among signatures of application methods in one of the application objects, and wherein the signatures of the application methods comprise public interfaces of the application methods;

automatically generating a user interface for the computer program, wherein automatically generating a user interface includes providing for at least one of a user and a programmer of the computer program to customize mappings between the application objects and user interface elements, wherein the user interface elements include at least one of user interface widgets and speech grammar rules, wherein the identified programming patterns include at least one of an undo pattern, a structure pattern, a graphical pattern, a validation pattern, and a precondition pattern, and wherein the undo pattern identifies, for each application method a pattern-based executed-command-object that can undo/redo the method, wherein the pattern-based executed-command-object comprises an executed-command object that uses at least one of;

(a) relationships among the signatures of the application methods in and (b) antonym dictionaries to implement the undo and redo operations, and wherein the executed-command object is an object that provides operations to undo and redo each application method.