Configurable speech recognition system using multiple recognizers

• US 8,898,065 B2
• Filed: 01/06/2012
• Issued: 11/25/2014
• Est. Priority Date: 01/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Newman, Michael, Gillet, Anthony, Krowitz, David Mark, Edgington, Michael D.
• Primary Examiner(s): Abebe, Daniel D
• Application Number:
  US13/345,265
• Publication Number:
  US 20120179471A1
• Time in Patent Office: 1,054 Days
• Field of Search: 704/275
• US Class Current: 704/275
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

Techniques for combining the results of multiple recognizers in a distributed speech recognition architecture. Speech data input to a client device is encoded and processed both locally and remotely by different recognizers configured to be proficient at different speech recognition tasks. The client/server architecture is configurable to enable network providers to specify a policy directed to a trade-off between reducing recognition latency perceived by a user and usage of network resources. The results of the local and remote speech recognition engines are combined based, at least in part, on logic stored by one or more components of the client/server architecture.

Claims

1. A method of performing speech recognition in a distributed system comprising an electronic device including an embedded speech recognizer and a network device including a remote speech recognizer remote from the electronic device, the method comprising:

receiving, by the electronic device, input audio comprising speech;

determining that at least a portion of the input audio matches a recognition grammar associated with the embedded speech recognizer;

generating a search tree associated with the recognition grammar, wherein the search tree includes a plurality of nodes, wherein each of the nodes is associated with a type of item in the recognition grammar;

determining whether the recognition grammar includes at least one generic speech nod, wherein determining whether the recognition grammar includes at least one generic speech node comprises determining whether the search tree includes only nodes associated with types of items that can be recognized by the embedded speech recognizer with an accuracy above a threshold;

determining that recognition by the remote speech recognizer is desired in response to determining that the recognition grammar includes at least one generic speech node indicating that the speech in the input audio may include free-form dictation; and

sending at least a portion of the input audio to the network device in response to determining that recognition by the remote speech recognizer is desired.

2. The method of claim 1, further comprising:

closing a network connection to the network device in response to determining that recognition by the remote speech recognizer is not desired.

3. The method of claim 1, further comprising:

determining that at least one element of the recognition grammar has not been completed based on an analysis of the input speech;

determining whether any of the uncompleted elements of the recognition grammar includes at least one generic speech node; and

transmitting at least a portion of the input audio to the network device in response to determining that at least one of the uncompleted elements in the recognition grammar includes at least one generic speech node.

4. The method of claim 1, wherein determining that at least a portion of the input audio matches a recognition grammar comprises determining that the at least a portion of the input audio matches a voice command.

5. The method of claim 4, wherein the voice command relates to performing a web-based search and wherein the recognition grammar is a search grammar that includes at least one generic speech node corresponding to web-search parameters.

6. The method of claim 1, further comprising:

receiving additional input audio comprising speech;

updating at least one node in the recognition grammar based on the additional input audio; and

re-evaluating the recognition grammar to determine if any of the nodes in the recognition grammar includes at least one generic speech node.

7. The method of claim 6, further comprising:

closing a network connection to the network device in response to determining that none of the nodes in the re-evaluated recognition grammar includes at least one generic speech node.

8. The method of claim 1, wherein the at least one generic speech node corresponds to input audio that may include free-flow dictation and/or web-search parameters.

9. A method of performing speech recognition in a distributed system comprising an electronic device including an embedded speech recognizer and a network device including a remote speech recognizer remote from the electronic device, the method comprising:

receiving, by the electronic device, input audio comprising speech;

determining that at least a portion of the input audio matches a recognition grammar associated with the embedded speech recognizer, wherein determining that at least a portion of the input audio matches a recognition grammar comprises determining that the at least a portion of the input audio matches a voice command for sending a text message, and wherein the recognition grammar is a text message grammar for the voice command that includes at least one generic speech node corresponding to text inserted into a body of the text message;

sending at least a portion of the input audio to the network device in response to determining that the at least a portion of the input audio matches the voice command for sending a text message.

10. An electronic device for use in a distributed speech recognition system comprising the electronic device and a network device remote from the electronic device, the electronic device, comprising:

an embedded speech recognizer configured to receive input audio comprising speech; and

at least one processor programmed to;

determine that at least a portion of the input audio matches a recognition grammar associated with the embedded speech recognizer;

generate a search tree associated with the recognition grammar, wherein the search tree includes a plurality of nodes, wherein each of the nodes is associated with a type of item in the recognition grammar;

determine whether the recognition grammar includes at least one generic speech node, wherein determining whether the recognition grammar includes at least one generic speech node comprises determining whether the search tree includes only nodes associated with types of items that can be recognized by the embedded speech recognizer with an accuracy above a threshold;

determine that recognition by the remote speech recognizer is desired in response to determining that the recognition grammar includes at least one generic speech node indicating that the speech in the input audio may include free-form dictation; and

send at least a portion of the input audio to the network device in response to determining that recognition by the remote speech recognizer is desired.

11. The electronic device of claim 10, wherein the at least one processor is further programmed to:

determine that at least one element of the recognition grammar has not been completed based on an analysis of the input speech;

determine whether any of the uncompleted elements of the recognition grammar includes at least one generic speech node; and

transmit at least a portion of the input audio to the network device in response to determining that at least one of the uncompleted elements in the recognition grammar includes at least one generic speech node.

12. The electronic device of claim 10, wherein the at least one processor is further programmed to close a network connection to the network device in response to determining that recognition by the remote speech recognizer is not desired.

13. The electronic device of claim 10, wherein determining that at least a portion of the input audio matches a recognition grammar comprises determining that the at least a portion of the input audio matches a voice command.

14. The electronic device of claim 13, wherein the voice command relates to sending a text message and wherein the recognition grammar is a text message grammar that includes at least one generic speech node corresponding to text inserted into a body of the text message.

15. The electronic device of claim 10, wherein the at least one processor is further programmed to:

update at least one node in the recognition grammar based on the additional input audio received by the embedded speech recognizer; and

re-evaluate the recognition grammar to determine if any of the nodes in the recognition grammar includes at least one generic speech node.

16. The electronic device of claim 15, wherein the at least one processor is further programmed to close a network connection to the network device in response to determining that none of the nodes in the re-evaluated recognition grammar includes at least one generic speech node.

17. The electronic device of claim 10, wherein the at least one generic speech node corresponds to input audio that may include free-flow dictation and/or web-search parameters.

18. An electronic device for use in a distributed speech recognition system comprising the electronic device and a network device remote from the electronic device, the electronic device, comprising:

an embedded speech recognizer configured to receive input audio comprising speech; and

at least one processor programmed to;

determine that at least a portion of the input audio matches a recognition grammar associated with the embedded speech recognizer, wherein determining that at least a portion of the input audio matches a recognition grammar comprises determining that at least a portion of the input audio matches a voice command for sending a text message, and wherein the recognition grammar is a text message grammar for the voice command that includes at least one generic speech node corresponding to text inserted into a body of the text message; and

send at least a portion of the input audio to the network device in response to determining that at least a portion of the input audio matches the voice command for sending a text message.