Adding speech capabilities to existing computer applications with complex graphical user interfaces

• US 9,081,550 B2
• Filed: 04/26/2011
• Issued: 07/14/2015
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Muszta, Attila, Neubacher, Andreas, Kozma, Peter, z, Mert, Hle, Herwig
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): Sirjani, Fariba
• Application Number:
  US13/094,313
• Publication Number:
  US 20120215543A1
• Time in Patent Office: 1,540 Days
• Field of Search: 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

At design time of a graphical user interface (GUI), a software component (VUIcontroller) is added to the GUI. At run time of the GUI, the VUIcontroller analyzes the GUI from within a process that executes the GUI. From this analysis, the VUIcontroller automatically generates a voice command set, such as a speech-recognition grammar, that corresponds to controls of the GUI. The generated voice command set is made available to a speech recognition engine, thereby speech-enabling the GUI. Optionally, a GUI designer may add properties to ones of the GUI controls at GUI design time, without necessarily writing a voice command set. These properties, if specified, are then used at GUI run time to control or influence the analysis of the GUI and the automatic generation of the voice command set.

Claims

1. Non-transitory computer-readable medium comprising computer-readable instructions that, when executed by a computer, adapt the computer to:

provide a voice user interface controller (VUIcontroller) for automatically speech enabling a graphical user interface (GUI) at run time of the GUI;

configure a user interface of the computer to receive first input from a GUI designer that identifies and includes the VUIcontroller in the GUI while designing the GUI so that the VUIcontroller will execute at run time of the GUI;

configure the user interface of the computer to receive second input from the GUI designer that identifies and assigns, at design time of the GUI, at least one voice-enabling property to a first GUI control of the GUI;

configure the user interface of the computer to receive third input from the GUI designer that identifies and disables, at design time of the GUI, voice control of a second GUI control of the GUI, wherein the VUIcontroller comprises instructions that are automatically executed at run time of the GUI to automatically;

analyze the GUI from within a process that executes the GUI;

generate a voice-enabled GUI based on the analysis;

generate, from the analysis, a voice command set corresponding to voice-enabled controls of the GUI; and

make the generated voice command set available to a speech-recognition engine.

2. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions, when executed, further adapt the computer to provide a drag-and-drop tool for use in a visual GUI integrated design environment (IDE) to include the VUIcontroller in the GUI.

3. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions for the VUIcontroller comprise a routine callable from a GUI markup document.

4. The non-transitory computer-readable medium of claim 3, wherein the computer-readable instructions are stored on a server other than a computer that renders the GUI at run time of the GUI, wherein the computer that renders the GUI and the server are interconnected, at least in part, by the Internet.

5. The non-transitory computer-readable medium of claim 4, wherein at least a portion of the analysis of the GUI is performed by the server.

6. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions for the VUIcontroller, when executed, further adapt the computer to automatically analyze the GUI by constructing a data structure representing the controls of the GUI.

7. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions for the VUIcontroller, when executed, further adapt the computer to respond, at run time of the GUI, to dynamic changes in the GUI by revising the generated voice command set according to the changes in the GUI.

8. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions for the VUIcontroller, when executed, further adapt the computer to respond, at run time of the GUI, to dynamic changes in the GUI by revising the generated voice command set according to the changes in the GUI and according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

9. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions for the VUIcontroller, when executed, further adapt the computer to respond, at run time of the GUI, to dynamic changes in the GUI by generating a revised voice command set according to the changed GUI.

10. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions for the VUIcontroller, when executed, further adapt the computer to respond, at run time of the GUI, to dynamic changes in the GUI by generating a revised voice command set according to the changed GUI and according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

11. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions, when executed, further adapt the computer to augment, at design time of the GUI, each of the controls of the GUI with at least one corresponding user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

12. The non-transitory computer-readable medium of claim 11, wherein the at least one user-settable property comprises text of a voice command for navigating to the corresponding control of the GUI.

13. The non-transitory computer-readable medium of claim 11, wherein the at least one user-settable property comprises an indicator of whether the corresponding control of the GUI is to be speech enabled.

14. The non-transitory computer-readable medium of claim 11, wherein:

the at least one user-settable property comprises a unique identifier associated with the control of the GUI; and

the computer-readable instructions, when executed, further adapt the computer to store, at design time of the GUI, an identifier of a user-specified speech recognition grammar associated with the unique identifier.

15. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions, when executed, further adapt the computer to augment, at design time of the GUI, each of the controls of the GUI with a unique identifier usable to identify an associated speech recognition grammar.

16. The non-transitory computer-readable medium of claim 1, wherein the automatic analysis of the GUI, when executed, is performed according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

17. The non-transitory computer-readable medium of claim 1, wherein the automatic generation of the voice command set is performed according to at least one user settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

18. The non-transitory computer-readable medium of claim 1, wherein the automatic making available of the generated voice command set to the speech-recognition engine is performed according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

19. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions, when executed, further adapt the computer to enable the GUI designer to specify more than one voice command for a GUI control.

20. The non-transitory computer-readable medium of claim 1, wherein the computer-readable instructions, when executed, further adapt the computer to enable the GUI designer to identify a predefined grammar for a GUI control.

21. A method for automatically speech enabling a graphical user interface (GUI), the method comprising:

at design time of the GUI, receiving first input from a GUI designer for design of the GUI;

at design time of the GUI, including a VUIcontroller in the GUI that will be executed at run time of the GUI;

at design time of the GUI, receiving second input the GUI designer that specifies a voice-enabling property of a first control of the GUI being designed;

at design time of the GUI, receiving third input from the GUI designer that disables voice control of a second control of the GUI being designed;

at run time of the GUI, automatically executing the VUIcontroller that was included in the GUI at design time of the GUI, wherein the VUIcontroller instantiates acts of;

analyzing the GUI from within a process that executes the GUI;

generating a voice-enabled GUI based on the analysis;

generating, from the analysis of the GUI, a voice command set corresponding to voice-enabled controls of the GUI; and

making the generated voice command set available to a speech-recognition engine.

22. A method according to claim 21, further comprising providing the VUIcontroller in a form of a drag-and-drop tool configured to be added to a visual GUI integrated development environment (IDE).

23. A method according to claim 21, wherein analyzing the GUI comprises analyzing the GUI according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

24. A method according to claim 21, wherein generating the voice command set comprises generating the voice command set according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

25. A method according to claim 21, wherein making the generated voice command set available to the speech-recognition engine comprises making the generated voice command set available to the speech-recognition engine according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

26. A method according to claim 21, further comprising, at run time of the GUI, in response to a change in the GUI, revising the generated voice command set according to the changes in the GUI.

27. A method according to claim 21, further comprising, at run time of the GUI, in response to a change in the GUI, revising the generated voice command set according to the changes in the GUI according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

28. A method according to claim 21, further comprising, at run time of the GUI, in response to a change in the GUI, generating a revised voice command set according to the changes in the GUI.

29. A method according to claim 21, further comprising, at run time of the GUI, in response to a change in the GUI, generating a revised voice command set according to the changes in the GUI according to at least one user-settable property, the user-settable property being settable to a value by the GUI designer at design time of the GUI.

30. A data processing system comprising:

a display; and

a processor adapted with computer-readable instructions to;

include a voice user interface controller (VUIcontroller) in a graphical user interface (GUI) at design time of the GUI that will be executed at run time of the GUI;

receive, at design time of the GUI, first input from a GUI designer that specifies a voice-enabling property of a first selected control of the GUI being designed;

receive, at design time of the GUI, second input from the GUI designer that disables voice control of a second selected control of the GUI being designed;

automatically execute the VUIcontroller at run time of the GUI;

automatically analyze, by the VUIcontroller at run time, the GUI from within a process that executes the GUI;

generate a voice-enabled GUI based on the analysis;

generate, from the analysis of the GUI, a voice command set corresponding to voice-enabled controls of the GUI; and

make the generated voice command set available to a speech recognition engine.