System and method for using prosody for voice-enabled search

• US 10,002,608 B2
• Filed: 09/17/2010
• Issued: 06/19/2018
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Bangalore, Srinivas, Feng, Junlan, Johnston, Michael, Mishra, Taniya
• Primary Examiner(s): WOZNIAK, JAMES S
• Application Number:
  US12/884,959
• Publication Number:
  US 20120072217A1
• Time in Patent Office: 2,832 Days
• Field of Search: 704207, 704251, 704257, 704270, 704275, 707706, 707748
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/54 for retrieval
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for approximating relevant responses to a user query with voice-enabled search. A system practicing the method receives a word lattice generated by an automatic speech recognizer based on a user speech and a prosodic analysis of the user speech, generates a reweighted word lattice based on the word lattice and the prosodic analysis, approximates based on the reweighted word lattice one or more relevant responses to the query, and presents to a user the responses to the query. The prosodic analysis examines metalinguistic information of the user speech and can identify the most salient subject matter of the speech, assess how confident a speaker is in the content of his or her speech, and identify the attitude, mood, emotion, sentiment, etc. of the speaker. Other information not described in the content of the speech can also be used.

Claims

1. A method comprising:

receiving a word lattice generated by an automatic speech recognizer processing a query, wherein the word lattice is weighted according to the query;

identifying a policy which allows use of a user emotional state in responding to a user who produced the query;

performing, via a processor of the automatic speech recognizer, a prosodic analysis of the query, wherein the prosodic analysis identifies an audible gesture in the query and a rhythm of words spoken in the query;

identifying, according to the prosodic analysis, the user emotional state;

reweighting, via the processor, the word lattice according to the prosodic analysis, the user emotional state and one of a time of day, a time of year, and a behavioral history of the user, to yield a reweighted word lattice;

determining, via the processor and according to the reweighted word lattice, a response to the query, the response addressing the audible gesture; and

presenting to the user the response to the query.

2. The method of claim 1, wherein the prosodic analysis further examines metalinguistic information of the query according to one or more of a tonality, a volume, a stress, an intonation, and a speed of the query.

3. The method of claim 2, wherein the prosodic analysis identifies a most salient subject matter of the query.

4. The method of claim 2, wherein the prosodic analysis assesses a confidence of the user in content of the query.

5. The method of claim 2, wherein the prosodic analysis identifies a mood of the user.

6. The method of claim 1, wherein determining the response is performed by composing the reweighted word lattice with a search finite state transducer according to a plurality of pre-indexed documents.

7. The method of claim 6, further comprising generating a finite state transducer according to composing the reweighted word lattice.

8. The method of claim 1, wherein the audible gesture comprises one of a laugh, a sob, a cough, and a yawn.

9. A system comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

receiving a word lattice generated by an automatic speech recognizer a query, wherein the word lattice is weighted according to the query;

identifying a policy which allows use of a user emotional state in responding to a user who produced the query;

performing a prosodic analysis of the query, wherein the prosodic analysis identifies an audible gesture in the query and a rhythm of words spoken in the query;

identifying, according to the prosodic analysis, the user emotional state;

reweighting the word lattice according to the prosodic analysis, the user emotional state and one of a time of day, a time of year, and a behavioral history of the user to yield a reweighted word lattice;

determining, according to the reweighted word lattice, a response to the query, the response addressing the audible gesture; and

presenting to the user the response to the query.

10. The system of claim 9, wherein the prosodic analysis examines metalinguistic information of the query according to one or more of a tonality, a volume, a stress, an intonation, and a speed of the query.

11. The system of claim 10, wherein the prosodic analysis identifies a most salient subject matter of the query.

12. The system of claim 10, wherein the prosodic analysis assesses a confidence of the user in content of the query.

13. The system of claim 10, wherein the prosodic analysis identifies a mood of the user.

14. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

receiving a word lattice generated by an automatic speech recognizer processing a query, wherein the word lattice is weighted according to the query;

identifying a policy which allows use of a user emotional state in responding to a user who produced the query;

performing a prosodic analysis of the query, wherein the prosodic analysis identifies an audible gesture in the query and a rhythm of words spoken in the query;

identifying, according to the prosodic analysis, the user emotional state;

reweighting the word lattice according to the prosodic analysis, the user emotional state and one of a time of day, a time of year, and a behavioral history of the user, to yield a reweighted word lattice;

determining, according to the reweighted word lattice, a response to the query, the response addressing the audible gesture; and

presenting to the user the response to the query.

15. The computer-readable storage device of claim 14, wherein determining the response is performed by composing the reweighted word lattice with a search finite state transducer according to a plurality of pre-indexed documents.

16. The computer-readable storage device of claim 15, having additional instructions stored which, when executed by the computing device, result in operations comprising generating a finite state transducer according to composing the reweighted word lattice.

17. The computer-readable storage device of claim 16, having additional instructions stored which, when executed by the computing device, result in operations comprising reranking, according to the finite state transducer, the word lattice generated by the automatic speech recognizer.