Multi-modal conversion tool for form-type applications

• US 8,352,908 B2
• Filed: 06/28/2010
• Issued: 01/08/2013
• Est. Priority Date: 06/28/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jhoney, Albee, Vaidyanathan, Praveen Kumar
• Primary Examiner(s): DENG, ANNA CHEN
• Application Number:
  US12/824,408
• Publication Number:
  US 20110321008A1
• Time in Patent Office: 925 Days
• Field of Search: 717106-109
• US Class Current: 717/106
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 8/74 Reverse engineering; Extracting design information from source code
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

GUI form code comprising a set of GUI elements can be imported. A user interface description can be generated from the GUI form code that has an element corresponding to each GUI element. For each user interface element converted from a corresponding to one of the GUI elements, a user interface element type can be determined as can temporal associations between the user interface elements. A Conversation User Interface (CUI) code corresponding to the GUI form code can be created from the user interface description. When creating the CUI code for each of the user interface elements, different and rules to convert the user interface element into CUI code can be used depending on a user interface element type of the user interface element being converted. When creating the CUI code, the user interface elements can be temporally ordered based on the pre-determined spatio-temporal associations between the graphical user interface (GUI) elements.

Claims

1. A method for generating a Conversation User

Interface (CUI) from a graphical user interface (GUI) comprising:

importing GUI form code comprising a plurality of GUI elements;

generating a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code;

for each of user interface element converted from a corresponding to one of the GUI elements, determining a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types;

for each user interface element, determining temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code;

determining whether disambiguation is needed for any of the GUI elements when generating the user interface description;

when no disambiguation is needed, automatically generating the user interface description from the GUI form code without any user input;

when disambiguation is needed, prompting a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description;

creating Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality;

when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilizing a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, and

when creating the Conversation User Interface (CUI) code, temporally ordering the user interface elements based on the determined temporal associations between the user interface elements.

2. The method of claim 1, wherein the user interface description corresponds to the GUI form code, but is language independent, where the GUI form code is specific to a programming language.

3. The method of claim 2, wherein the conversion from GUI form code to the user interface description is performed by executing an injective function that preserves distinctness in that every element of the injective function'"'"'s codomain is mapped to by at most one element of the injective function'"'"'s domain, wherein executing the injective function against different GUI form code segments that are substantially equivalent to each other yet written in different programming languages results in the same user interface description being generated.

4. The method of claim 1, wherein the GUI form code natively lacks conversation user interface specific elements, which are generated automatically by programmatic routines without manual user coding for the conversation user interface code being required.

5. The method of claim 1, wherein the voice interactive modality uses the conversation user interface code to accept and process speech input and to generate speech output, where the speech input and speech output is synchronized with GUI elements of the GUI form code.

6. The method of claim 5, wherein the visual modality of the multi-modal user interface comprises a chat section within which the speech input is visually presented as text and within which speech output is visually presented as text, wherein the speech input and speech output are handled in accordance with the conversation user interface (CUI) code.

7. The method of claim 1, wherein the determined temporal associations between the user interface elements is determined based on a spatial orientation of corresponding GUI elements of the GUI form code, of parent-child relationships between the corresponding GUI elements in the GUI form code, and of tab order of the corresponding GUI elements in the GUI form code.

8. The method of claim 1, wherein said user interface element type is determined from a set of user interface element types comprising an association type for an assertion element, an action type for an action element, an interrogator type for an interrogatory element, a container type for a container element, and a display type for a display element.

9. The method of claim 8, wherein the determined user interface element type for at least one of the user interface elements is for an assertion element.

10. The method of claim 8, wherein the determined user interface element type for at least one of the user interface elements is for an action element.

11. The method of claim 8, wherein the determined user interface element type for at least one of the user interface elements is for an interrogatory element.

12. The method of claim 8, wherein the determined user interface element type for at least one of the user interface elements is for a container element.

13. The method of claim 8, wherein the determined user interface element type for at least one of the user interface elements is for a display element.

14. The method of claim 1, further comprising:

transforming the user interface description into a conversation user interface (CUI) flow, wherein said conversation user interface (CUI) flow comprises a plurality of nodes and edges, each node representing a task, each edge representing a temporal flow in a dialog, wherein the conversation user interface (CUI) code is generated from the conversation user interface (CUI) flow.

15. The method of claim 1, further comprising:

generating a Conversation User Interface (CUI) activity tree from the user interface description;

annotating the activity tree with dialog fragments;

generating Conversation User Interface (CUI) rendering scripts, Conversation User Interface (CUI) form speech interpreting grammar entries, and Conversation User Interface (CUI) conversational scripts from the annotated activity tree, wherein the Conversation User Interface (CUI) code includes the Conversation User Interface (CUI) rendering scripts, the Conversation User Interface (CUI) form speech interpreting grammar entries, and the Conversation User Interface (CUI) conversational scripts.

16. The method of claim 15, wherein the Conversation User Interface (CUI) rendering scripts are each scripts used by a Conversation User Interface (CUI) run-time to textually render contents of the GUI elements, wherein the conversation form speech recognition grammar is a script used by a CUI run-time to interpret user input provided as speech, and wherein the CUI form conversation script is a script used by a CUI run-time to prompt a user for input and to textually render error messages.

17. A computer program product comprising a non-transitory computer readable storage medium having computer usable program code embodied therewith, the computer usable program code comprising:

computer usable program code stored in a non-transitory storage medium operable to import GUI form code comprising a plurality of GUI elements;

computer usable program code stored in a non-transitory storage medium operable to generate a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code;

computer usable program code stored in a non-transitory storage medium operable to, for each of user interface element converted from a corresponding to one of the GUI elements, determine a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types;

computer usable program code stored in a non-transitory storage medium operable to, for each user interface element, determine temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code;

computer usable program code stored in a non-transitory storage medium operable to determine whether disambiguation is needed for any of the GUI elements when generating the user interface description;

computer usable program code stored in a non-transitory storage medium operable to, when no disambiguation is needed, automatically generate the user interface description from the GUI form code without any user input;

computer usable program code stored in a non-transitory storage medium operable to, when disambiguation is needed, prompt a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description;

computer usable program code stored in a non-transitory tangible storage medium operable to create Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality;

computer usable program code stored in a non-transitory storage medium operable to, when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilize a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, and

computer usable program code stored in a tan non-transitory storage medium operable to, when creating the Conversation User Interface (CUI) code, temporally order the user interface elements based on the determined temporal associations between the user interface elements.

18. A system for generating a Conversation User Interface (CUI) from a graphical user interface (GUI) comprising:

a processor;

a non-volatile data store;

a communication bus communicatively connecting the processor and the non-volatile data store to each other; and

at least one computer program product digitally encoded within the non-volatile data store and operable to be executed by the processor, wherein running the computer program product within the system causes the system to;

import GUI form code comprising a plurality of GUI elements;

generate a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code;

for each of user interface element converted from a corresponding to one of the GUI elements, determine a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types;

for each user interface element, determine temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code;

determine whether disambiguation is needed for any of the GUI elements when generating the user interface description;

when no disambiguation is needed, automatically generate the user interface description from the GUI form code without any user input;

when disambiguation is needed, prompt a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description;

create Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality;

when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilize a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, and

when creating the Conversation User Interface (CUI) code, temporally order the user interface elements based on the determined temporal associations between the user interface elements.