Displaying speech command input state information in a multimodal browser

• US 8,965,772 B2
• Filed: 03/20/2014
• Issued: 02/24/2015
• Est. Priority Date: 09/13/2005
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Cross, Charles W. Jr., Hollinger, Michael C., Jablokov, Igor R., Lewis, Benjamin D., Pike, Hilary A., Smith, Daniel M., Wintermute, David W., Zaitzeff, Michael A.
• Primary Examiner(s): Serrou, Abdelali
• Application Number:
  US14/220,281
• Publication Number:
  US 20140208210A1
• Time in Patent Office: 341 Days
• Field of Search: 704/270, 704/275, 704/270.1, 704/271, 704/E15.04, 704/E15.047, 704/235, 704/E15.044, 704/231, 704/251, 704/257, 704/E15.045, 704/E15.001, 704/9
• US Class Current: 704/275
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Methods, systems, and products are disclosed for displaying speech command input state information in a multimodal browser including displaying an icon representing a speech command type and displaying an icon representing the input state of the speech command. In typical embodiments, the icon representing a speech command type and the icon representing the input state of the speech command also includes attributes of a single icon. Typical embodiments include accepting from a user a speech command of the speech command type, changing the input state of the speech command, and displaying another icon representing the changed input state of the speech command. Typical embodiments also include displaying the text of the speech command in association with the icon representing the speech command type.

Claims

1. A method for operating a computing system having at least one processor and a display, comprising acts of:

displaying on the display a first input widget via which text of user speech commands of a first speech command type can be presented, the text of the user speech commands presented via the first input widget being identified by evaluating received user speech commands against a first grammar associated with the first speech command type;

displaying on the display a first icon that allows a user to determine that the first input widget is currently capable of being used to input speech commands of the first speech command type to an application;

displaying on the display a second input widget via which text of user speech commands of a second speech command type can be presented, the text of the user speech commands presented via the second input widget being identified by evaluating received user speech commands against a second grammar, which is different than the first grammar, associated with the second speech command type;

displaying on the display a second icon that allows the user to determine that the second input widget is currently capable of being used to input speech commands of the second speech command type to the application;

receiving a user speech command;

with the at least one processor, determining whether the received user speech command is of the first speech command type;

if the first input widget is currently capable of being used to input speech commands of the first speech command type to the application and the at least one processor determines the received user speech command is of the first speech command type, then causing the display to present the text of the received user speech command of the first speech command type via the first input widget;

altering the first icon displayed on the display so as to allow the user to determine that the first input widget is currently not capable of being used to input speech commands of the first speech command type to the application;

with the at least one processor, determining whether the received user speech command is of the second speech command type;

if the second input widget is currently capable of being used to input speech commands of the second speech command type to the application and the at least one processor determines the received user speech command is of the second speech command type, then causing the display to present the text of the received user speech command of the second speech command type via the second input widget; and

altering the second icon displayed on the display so as to allow the user to determine that the second input widget is currently not capable of being used to input speech commands of the second speech command type to the application.

2. The method of claim 1, further comprising acts of:

displaying on the display in association with the first input widget first information that allows the user to identify the first type of speech command the first input widget is configured to accept; and

displaying on the display in association with the second input widget second information that allows the user to identify the second type of speech command the second input widget is configured to accept.

3. The method of claim 2, wherein the first icon allows the user to ascertain the first information and comprises an attribute that allows the user to determine whether the first input widget is currently capable of being used to input speech commands of the first speech command type to the application.

4. The method of claim 3, wherein the attribute of the first icon comprises a color of at least a portion of the first icon.

5. The method of claim 1, wherein the application comprises a multimodal application and the method further comprises an act of:

accepting inputs to the multimodal application from one or both of the first and second input widgets via at least one mode other than speech.

6. The method of claim 1, wherein:

the act of determining whether the received user speech command is of the first speech command type comprises determining whether the received user speech command matches an entry in the first grammar; and

the act of determining whether the received user speech command is of the second speech command type comprises determining whether the received user speech command matches an entry in the second grammar.

7. An apparatus comprising at least one processor and a display, wherein the at least one processor is configured to:

cause the display to display a first input widget via which text of user speech commands of a first speech command type can be presented, the apparatus being configured such that the text of the user speech commands presented via the first input widget is identified by evaluating received user speech commands against a first grammar associated with the first speech command type;

cause the display to display a first icon that allows a user to determine that the first input widget is currently capable of being used to input speech commands of the first speech command type to an application;

cause the display to display a second input widget via which text of user speech commands of a second speech command type can be presented, the apparatus being configured such that the text of the user speech commands presented via the second input widget is identified by evaluating received user speech commands against a second grammar, which is different than the first grammar, associated with the second speech command type;

cause the display to display a second icon that allows the user to determine that the second input widget is currently capable of being used to input speech commands of the second speech command type to the application;

receive a user speech command;

determine whether the received user speech command is of the first speech command type;

if the first input widget is currently capable of being used to input speech commands of the first speech command type to the application and the at least one processor determines the received user speech command is of the first speech command type, cause the display to present the text of the received user speech command of the first speech command type via the first input widget;

alter the first icon displayed on the display so as to allow the user to determine that the first input widget is currently not capable of being used to input speech commands of the first speech command type to the application;

determine whether the received user speech command is of the second speech command type;

if the second input widget is currently capable of being used to input speech commands of the second speech command type to the application and the at least one processor determines the received user speech command is of the second speech command type, cause the display to present the text of the received user speech command of the second speech command type via the second input widget; and

alter the second icon displayed on the display so as to allow the user to determine that the second input widget is currently not capable of being used to input speech commands of the second speech command type to the application.

8. The apparatus of claim 7, wherein the at least one processor is further configured to:

display on the display in association with the first input widget first information that allows the user to identify the first type of speech command the first input widget is configured to accept; and

display on the display in association with the second input widget second information that allows the user to identify the second type of speech command the second input widget is configured to accept.

9. The apparatus of claim 8, wherein the first icon allows the user to ascertain the first information and comprises an attribute that allows the user to determine whether the first input widget is currently capable of being used to input speech commands of the first speech command type to the application.

10. The apparatus of claim 9, wherein the attribute of the first icon comprises a color of at least a portion of the first icon.

11. The apparatus of claim 7, wherein the application comprises a multimodal application and the at least one processor is configured such that one or both of the first and second input widgets can alternatively accept inputs via at least one mode other than speech.

12. The apparatus of claim 7, wherein the at least one processor is configured to:

determine whether the received user speech command is of the first speech command type comprises by determining whether the received user speech command matches an entry in the first grammar; and

determine whether the received user speech command is of the second speech command type comprises by determining whether the received user speech command matches an entry in the second grammar.

13. A non-transitory computer-readable medium encoded with instructions which, when executed by at least one processor, cause the at least one processor to perform a method comprising acts of:

displaying on a display a first input widget via which text of user speech commands of a first speech command type can be presented, the text of the user speech commands presented via the first input widget being identified by evaluating received user speech commands against a first grammar associated with the first speech command type;

displaying on the display a first icon that allows a user to determine that the first input widget is currently capable of being used to input speech commands of the first speech command type to an application;

displaying on the display a second input widget via which text of user speech commands of a second speech command type can be presented, the text of the user speech commands presented via the second input widget being identified by evaluating received user speech commands against a second grammar, which is different than the first grammar, associated with the second speech command type;

displaying on the display a second icon that allows the user to determine that the second input widget is currently capable of being used to input speech commands of the second speech command type to the application;

receiving a user speech command;

determining whether the received user speech command is of the first speech command type;

if the first input widget is currently capable of being used to input speech commands of the first speech command type to the application and the at least one processor determines the received user speech command is of the first speech command type, then causing the display to present the text of the received user speech command of the first speech command type via the first input widget;

altering the first icon displayed on the display so as to allow the user to determine that the first input widget is currently not capable of being used to input speech commands of the first speech command type to the application;

determining whether the received user speech command is of the second speech command type;

if the second input widget is currently capable of being used to input speech commands of the second speech command type to the application and the at least one processor determines the received user speech command is of the second speech command type, then causing the display to present the text of the received user speech command of the second speech command type via the second input widget; and

altering the second icon displayed on the display so as to allow the user to determine that the second input widget is currently not capable of being used to input speech commands of the second speech command type to the application.

14. The computer-readable medium of claim 13, encoded with further instructions which, when executed by the at least one processor, cause the at least one processor to perform further acts of:

displaying on the display in association with the first input widget first information that allows the user to identify the first type of speech command the first input widget is configured to accept; and

displaying on the display in association with the second input widget second information that allows the user to identify the second type of speech command the second input widget is configured to accept.

15. The computer-readable medium of claim 14, wherein the first icon allows the user to ascertain the first information and comprises an attribute that allows the user to determine whether the first input widget is currently capable of being used to input speech commands of the first speech command type to the application.

16. The computer-readable medium of claim 15, wherein the attribute of the first icon comprises a color of at least a portion of the first icon.

17. The computer-readable medium of claim 13, wherein the application comprises a multimodal application and wherein and the computer-readable medium is further encoded with instructions which, when executed by the at least one processor, allow the multimodal application to alternatively accept inputs from one or both of the first and second input widgets via at least one mode other than speech.

18. The computer-readable medium of claim 13, wherein:

the act of determining whether the received user speech command is of the first speech command type comprises determining whether the received user speech command matches an entry in the first grammar; and

the act of determining whether the received user speech command is of the second speech command type comprises determining whether the received user speech command matches an entry in the second grammar.