Adding Speech Capabilities to Existing Computer Applications with Complex Graphical User Interfaces

• US 20120215543A1
• Filed: 04/26/2011
• Published: 08/23/2012
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Muszta, Attila, Neubacher, Andreas, Öz, Mert, Häle, Herwig, Kozma, Peter
• Granted Patent: US 9,081,550 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

At design time of a graphical user interface (GUI), a software component (VUIcontroller) is added to the GUI. At run time of the GUI, the VUIcontroller analyzes the GUI from within a process that executes the GUI. From this analysis, the VUIcontroller automatically generates a voice command set, such as a speech-recognition grammar, that corresponds to controls of the GUI. The generated voice command set is made available to a speech recognition engine, thereby speech-enabling the GUI. Optionally, a GUI designer may add properties to ones of the GUI controls at GUI design time, without necessarily writing a voice command set. These properties, if specified, are then used at GUI run time to control or influence the analysis of the GUI and the automatic generation of the voice command set.

Claims

1. A voice user interface controller (VUIcontroller) for automatically speech enabling a graphical user interface (GUI), comprising:

a software component configured for inclusion, at design time of the GUI, in the GUI, such that the software component is automatically executed at run time of the GUI, and the software component is further configured to, at run time of the GUI, automatically;

analyze the GUI from within a process that executes the GUI;

generate, from the analysis, a voice command set corresponding to controls of the GUI; and

make the generated voice command set available to a speech-recognition engine.

2. A VUIcontroller according to claim 1, wherein the software component comprises a drag-and-drop tool for use in a visual GUI integrated design environment (IDE).

3. A VUIcontroller according to claim 1, wherein the software component comprises a routine callable from a GUI markup document.

4. A VUIcontroller according to claim 3, wherein the software component is stored on a server other than a computer that renders the GUI at run time of the GUI, wherein the computer that renders the GUI and the server are interconnected, at least in part, by the Internet.

5. A VUIcontroller according to claim 4, wherein at least a portion of the analysis of the GUI is performed by the server.

6. A VUIcontroller according to claim 1, wherein the software component is configured to automatically analyze the GUI by constructing a data structure representing the controls of the GUI.

7. A VUIcontroller according to claim 1, wherein the software component is configured to respond, at run time of the GUI, to dynamic changes in the GUI by revising the generated voice command set according to the changes in the GUI.

8. A VUIcontroller according to claim 1, wherein the software component is configured to respond, at run time of the GUI, to dynamic changes in the GUI by revising the generated voice command set according to the changes in the GUI and according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

9. A VUIcontroller according to claim 1, wherein the software component is configured to respond, at run time of the GUI, to dynamic changes in the GUI by generating a revised voice command set according to the changed GUI.

10. A VUIcontroller according to claim 1, wherein the software component is configured to respond, at run time of the GUI, to dynamic changes in the GUI by generating a revised voice command set according to the changed GUI and according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

11. A VUIcontroller according to claim 1, wherein the software component is further configured to augment, at design time of the GUI, each of the controls of the GUI with at least one corresponding user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

12. A VUIcontroller according to claim 11, wherein the at least one user-settable property comprises text of a voice command for navigating to the corresponding control of the GUI.

13. A VUIcontroller according to claim 11, wherein the at least one user-settable property comprises an indicator of whether the corresponding control of the GUI is to be speech enabled.

14. A VUIcontroller according to claim 11, wherein:

the at least one user-settable property comprises a unique identifier associated with the control of the GUI; and

the software component is further configured to store, at design time of the GUI, an identifier of a user-specified speech recognition grammar associated with the unique identifier.

15. A VUIcontroller according to claim 1, wherein the software component is further configured to augment, at design time of the GUI, each of the controls of the GUI with a unique identifier usable to identify an associated speech recognition grammar.

16. A VUIcontroller according to claim 1, wherein the software component is further configured such that the automatic analysis of the GUI is performed according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

17. A VUIcontroller according to claim 1, wherein the software component is further configured such that the automatic generation of the voice command set is performed according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

18. A VUIcontroller according to claim 1, wherein the software component is further configured such that the automatic making available of the generated voice command set to the speech-recognition engine is performed according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

19. A method for automatically speech enabling a graphical user interface (GUI), the method comprising:

at run time of the GUI, automatically executing a VUIcontroller that was included in the GUI at design time of the GUI, the VUIcontroller therefore automatically;

analyzing the GUI from within a process that executes the GUI;

generating, from the analysis of the GUI, a voice command set corresponding to controls of the GUI; and

making the generated voice command set available to a speech-recognition engine.

20. A method according to claim 19, further comprising providing the VUIcontroller in a form of a drag-and-drop tool configured to be added to a visual GUI integrated development environment (IDE).

21. A method according to claim 19, wherein analyzing the GUI comprises analyzing the GUI according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

22. A method according to claim 19, wherein generating the voice command set comprises generating the voice command set according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

23. A method according to claim 19, wherein making the generated voice command set available to the speech-recognition engine comprises making the generated voice command set available to the speech-recognition engine according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

24. A method according to claim 19, further comprising, at run time of the GUI, in response to a change in the GUI, revising the generated voice command set according to the changes in the GUI.

25. A method according to claim 19, further comprising, at run time of the GUI, in response to a change in the GUI, revising the generated voice command set according to the changes in the GUI according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

26. A method according to claim 19, further comprising, at run time of the GUI, in response to a change in the GUI, generating a revised voice command set according to the changes in the GUI.

27. A method according to claim 19, further comprising, at run time of the GUI, in response to a change in the GUI, generating a revised voice command set according to the changes in the GUI according to at least one user-settable property, the user-settable property being settable to a value by a user at design time of the GUI.

28. A non-transitory computer-readable medium having computer readable program code stored thereon for automatically speech enabling a graphical user interface (GUI), the stored program code comprising:

program code for, at run time of the GUI, automatically executing a VUIcontroller that was included in the GUI at design time of the GUI;

program code for, as a consequence of the automatic execution of the VUIcontroller, automatically analyzing the GUI from within a process that executes the GUI;

program code for automatically generating, from the analysis of the GUI, a voice command set corresponding to controls of the GUI; and

program code for automatically making the generated voice command set available to a speech recognition engine.