Facilitating multimodal interaction with grammar-based speech applications

• US 8,126,715 B2
• Filed: 11/26/2008
• Issued: 02/28/2012
• Est. Priority Date: 11/26/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Paek, Timothy Seung Yoon
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US12/324,826
• Publication Number:
  US 20100131275A1
• Time in Patent Office: 1,189 Days
• Field of Search: 704/257
• US Class Current: 704/257
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Multimodal interaction with grammar-based speech applications may be facilitated with a device by presenting permissible phrases that are in-grammar based on acceptable terms that are in-vocabulary and that have been recognized from a spoken utterance. In an example embodiment, a spoken utterance having two or more terms is received. The two or more terms include one or more acceptable terms. An index is searched using the acceptable terms as query terms. From the searching of the index, permissible phrase(s) are produced that include the acceptable terms. The index is a searchable data structure that represents multiple possible grammar paths that are ascertainable based on acceptable values for each term position of a grammar-based speech application. The permissible phrase(s) are presented to a user as option(s) that may be selected to conduct multimodal interaction with the device.

Claims

1. One or more processor-accessible tangible storage media storing processor-executable instructions for facilitating multimodal interaction, wherein the processor-executable instructions, when executed by a processor, perform a method comprising:

receiving a spoken utterance including two or more terms, the two or more terms not being recognized as a permissible phrase that is in-grammar for a grammar-based speech application;

recognizing one or more acceptable terms that are in-vocabulary from among the two or more terms of the spoken utterance;

searching an index using the one or more acceptable terms as query terms;

producing, from the searching of the index, at least one permissible phrase that is in-grammar and that includes the one or more acceptable terms;

the index comprising a searchable data structure that represents multiple possible grammar paths that are ascertainable based on acceptable values for each term position of the grammar-based speech application; and

presenting at least one permissible phrase to a user as at least one option that may be selected to conduct multimodal interaction with the device.

2. The one or more processor-accessible tangible storage media as recited in claim 1, wherein the method comprises:

detecting if the user selects the at least one option corresponding to the at least one permissible phrase; and

if the user is detected to select the at least one option, implementing functionality associated with the at least one permissible phrase.

3. The one or more processor-accessible tangible storage media as recited in claim 1, wherein the method comprises:

traversing the multiple possible grammar paths of the grammar-based speech application based on the acceptable values for each term position of the grammar-based speech application;

ascertaining, from the traversing of the multiple possible grammar paths, phrases that are permissible in accordance with the grammar-based speech application; and

creating the index from the phrases that are permissible in accordance with the grammar-based speech application.

4. The one or more processor-accessible tangible storage media as recited in claim 1, wherein the grammar-based speech application comprises a command-and-control speech application;

and wherein types of terms of the grammar-based speech application include slots, keywords, and carrier phrases.

5. The one or more processor-accessible tangible storage media as recited in claim 1, wherein the method comprises:

tracking a frequency at which the user selects the at least one option corresponding to the at least one permissible phrase; and

during another operation, repeating the act of recognizing, the act of producing, or the act of presenting with regard to the tracked frequency at which the user has selected the at least one option corresponding to the at least one permissible phrase.

6. A computer-implemented method for facilitating multimodal interaction, the method comprising:

receiving a spoken utterance having two or more terms, the two or more terms including one or more acceptable terms;

searching an index using the one or more acceptable terms as query terms;

producing using a computer processor, from the searching of the index, at least one permissible phrase that includes the one or more acceptable terms;

the index comprising a searchable data structure that represents multiple possible grammar paths that are ascertainable based on acceptable values for each term position of a grammar-based speech application; and

presenting the at least one permissible phrase to a user as at least one option that may be selected to conduct multimodal interaction with the device.

7. The method as recited in claim 6, wherein receiving comprises:

receiving the spoken utterance via at least one microphone or one or more transmission media.

8. The method as recited in claim 6, further comprising:

recognizing from the spoken utterance that the one or more acceptable terms are in-vocabulary in accordance with the grammar-based speech application using a speech recognition processing mechanism; and

failing to recognize the spoken utterance as including a permissible phrase that is in-grammar in accordance with the grammar-based speech application.

9. The method as recited in claim 6, wherein presenting comprises:

presenting the at least one permissible phrase to the user as the at least one option visually via a display screen, wherein the at least one option is visually displayed as one or more of text, icons, or pictures;

or

presenting the at least one permissible phrase to the user as the at least one option aurally via a speaker.

10. The method as recited in claim 9, further comprising:

detecting if the user selects the at least one option corresponding to the at least one permissible phrase;

wherein the act of detecting comprises detecting if the user;

touches the at least one option on the display screen, clicks on the at least one option on the display screen, gestures with respect to the at least one option on the display screen, vocalizes the at least one option, or vocalizes a part of the at least one option that was not previously spoken.

11. The method as recited in claim 6, further comprising:

traversing the multiple possible grammar paths of the grammar-based speech application based on the acceptable values for each term position of the grammar-based speech application to ascertain phrases that are permissible in accordance with the grammar-based speech application; and

creating the index from the phrases that are permissible in accordance with the grammar-based speech application.

12. The method as recited in claim 11, wherein the traversing comprises:

for a current term position, assigning an acceptable value that is in-vocabulary for the grammar-based speech application;

iterating through each acceptable value for the current term position;

recursively continuing through each term position in accordance with the grammar-based speech application to identify the multiple possible grammar paths; and

ascertaining the phrases that are permissible in accordance with the grammar-based speech application from the identified multiple possible grammar paths and the acceptable values.

13. The method as recited in claim 6, wherein producing comprises:

retrieving from the index one or more retrieved acceptable terms as part of the at least one permissible phrase;

wherein the method further comprises;

for a forward case in which the one or more acceptable terms from the spoken utterance comprise one or more recognized acceptable terms that initiate the at least one permissible phrase, presenting the one or more recognized acceptable terms followed by the one or more retrieved acceptable terms.

14. The method as recited in claim 6, wherein producing comprises:

retrieving from the index one or more retrieved acceptable terms as part of the at least one permissible phrase;

wherein the method further comprises;

for a backward case in which the one or more acceptable terms from the spoken utterance comprise one or more recognized acceptable terms that do not initiate the at least one permissible phrase, presenting the one or more retrieved acceptable terms followed by the one or more recognized acceptable terms.

15. A device for facilitating multimodal interaction, the device comprising:

a processor;

a speech processing component to receive a spoken utterance having two or more terms, the two or more terms including one or more acceptable terms;

a grammar-based speech application including an index and a search unit;

the index comprising a searchable data structure that represents multiple possible grammar paths that are ascertainable based on acceptable values for each term position of the grammar-based speech application;

the search unit to search the index using the one or more acceptable terms as query terms and to produce, using the processor, at least one permissible phrase that includes the one or more acceptable terms; and

a feedback component to present the at least one permissible phrase to a user as at least one option that may be selected to conduct multimodal interaction with the device.

16. The device as recited in claim 15, wherein each of the two or more terms comprises one or more words, and the one or more acceptable terms comprise words that are in-vocabulary;

and wherein the at least one permissible phrase comprises at least one phrase that is in-grammar in accordance with the grammar-based speech application.

17. The device as recited in claim 15, wherein the device comprises a mobile device;

or wherein the device comprises at least part of a vehicle.

18. The device as recited in claim 15, wherein the feedback component is to detect if the user selects the at least one option corresponding to the at least one permissible phrase;

and

wherein the device further comprises;

at least one functionality component to implement functionality associated with the at least one permissible phrase if the user is detected to select the at least one option.

19. The device as recited in claim 15, wherein the grammar-based speech application is adapted to handle a forward case, in which the one or more acceptable terms from the spoken utterance comprise one or more recognized acceptable terms that are recognizable as initiating the at least one permissible phrase, by presenting the one or more recognized acceptable terms followed by one or more retrieved acceptable terms that are retrieved from the index during the search;

and

wherein the grammar-based speech application is adapted to handle a backward case, in which the one or more acceptable terms from the spoken utterance comprise one or more recognized acceptable terms that do not initiate the at least one permissible phrase, using a backoff grammar and by presenting one or more retrieved acceptable terms followed by the one or more recognized acceptable terms.

20. The device as recited in claim 15, wherein the speech processing component fails to recognize the spoken utterance as including a permissible phrase that is in-grammar for the grammar-based speech application;

and

wherein the grammar-based speech application is to traverse the multiple possible grammar paths of the grammar-based speech application based on the acceptable values for each term position of multiple term positions to ascertain phrases that are permissible in accordance with the grammar-based speech application; and

wherein the grammar-based speech application is to create the index from the phrases that are permissible in accordance with the grammar-based speech application.