Multi-modal conversion tool for form-type applications
First Claim
1. A method for generating a Conversation UserInterface (CUI) from a graphical user interface (GUI) comprising:
- importing GUI form code comprising a plurality of GUI elements;
generating a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code;
for each of user interface element converted from a corresponding to one of the GUI elements, determining a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types;
for each user interface element, determining temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code;
determining whether disambiguation is needed for any of the GUI elements when generating the user interface description;
when no disambiguation is needed, automatically generating the user interface description from the GUI form code without any user input;
when disambiguation is needed, prompting a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description;
creating Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality;
when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilizing a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, andwhen creating the Conversation User Interface (CUI) code, temporally ordering the user interface elements based on the determined temporal associations between the user interface elements.
1 Assignment
0 Petitions
Accused Products
Abstract
GUI form code comprising a set of GUI elements can be imported. A user interface description can be generated from the GUI form code that has an element corresponding to each GUI element. For each user interface element converted from a corresponding to one of the GUI elements, a user interface element type can be determined as can temporal associations between the user interface elements. A Conversation User Interface (CUI) code corresponding to the GUI form code can be created from the user interface description. When creating the CUI code for each of the user interface elements, different and rules to convert the user interface element into CUI code can be used depending on a user interface element type of the user interface element being converted. When creating the CUI code, the user interface elements can be temporally ordered based on the pre-determined spatio-temporal associations between the graphical user interface (GUI) elements.
-
Citations
18 Claims
-
1. A method for generating a Conversation User
Interface (CUI) from a graphical user interface (GUI) comprising: -
importing GUI form code comprising a plurality of GUI elements; generating a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code; for each of user interface element converted from a corresponding to one of the GUI elements, determining a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types; for each user interface element, determining temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code; determining whether disambiguation is needed for any of the GUI elements when generating the user interface description; when no disambiguation is needed, automatically generating the user interface description from the GUI form code without any user input; when disambiguation is needed, prompting a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description; creating Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality; when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilizing a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, and when creating the Conversation User Interface (CUI) code, temporally ordering the user interface elements based on the determined temporal associations between the user interface elements. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. A computer program product comprising a non-transitory computer readable storage medium having computer usable program code embodied therewith, the computer usable program code comprising:
-
computer usable program code stored in a non-transitory storage medium operable to import GUI form code comprising a plurality of GUI elements; computer usable program code stored in a non-transitory storage medium operable to generate a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code; computer usable program code stored in a non-transitory storage medium operable to, for each of user interface element converted from a corresponding to one of the GUI elements, determine a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types; computer usable program code stored in a non-transitory storage medium operable to, for each user interface element, determine temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code; computer usable program code stored in a non-transitory storage medium operable to determine whether disambiguation is needed for any of the GUI elements when generating the user interface description; computer usable program code stored in a non-transitory storage medium operable to, when no disambiguation is needed, automatically generate the user interface description from the GUI form code without any user input; computer usable program code stored in a non-transitory storage medium operable to, when disambiguation is needed, prompt a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description; computer usable program code stored in a non-transitory tangible storage medium operable to create Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality; computer usable program code stored in a non-transitory storage medium operable to, when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilize a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, and computer usable program code stored in a tan non-transitory storage medium operable to, when creating the Conversation User Interface (CUI) code, temporally order the user interface elements based on the determined temporal associations between the user interface elements.
-
-
18. A system for generating a Conversation User Interface (CUI) from a graphical user interface (GUI) comprising:
-
a processor; a non-volatile data store; a communication bus communicatively connecting the processor and the non-volatile data store to each other; and at least one computer program product digitally encoded within the non-volatile data store and operable to be executed by the processor, wherein running the computer program product within the system causes the system to; import GUI form code comprising a plurality of GUI elements; generate a user interface description from the GUI form code, where the user interface description comprises a user interface element for each of the GUI elements of the GUI form code; for each of user interface element converted from a corresponding to one of the GUI elements, determine a user interface element type for the user interface element, wherein said determined user interface element type is one of a plurality of different element types; for each user interface element, determine temporal associations between the user interface element and other ones of the user interface elements, wherein said temporal associations are dependent upon characteristics of the corresponding GUI element relative to other ones of the GUI elements within the GUI form code; determine whether disambiguation is needed for any of the GUI elements when generating the user interface description; when no disambiguation is needed, automatically generate the user interface description from the GUI form code without any user input; when disambiguation is needed, prompt a user for disambiguation input for each GUI element for which disambiguation is needed, receiving a user response to the prompting, disambiguating otherwise ambiguous GUI elements utilizing the user response when generating the user interface description; create Conversation User Interface (CUI) code corresponding to the GUI form code from the user interface description, wherein the conversation user interface (CUI) code and the graphical form code is integrated and synchronized to form multi-modal user interface code having a visual and a voice interactive modality; when creating the Conversation User Interface (CUI) code for each of the user interface elements of the user interface description, utilize a different and rules to convert the user interface element into Conversation User Interface (CUI) code depending on a user interface element type of the user interface element being converted, and when creating the Conversation User Interface (CUI) code, temporally order the user interface elements based on the determined temporal associations between the user interface elements.
-
Specification