Configurable speech recognition system using multiple recognizers

• US 9,953,653 B2
• Filed: 01/06/2012
• Issued: 04/24/2018
• Est. Priority Date: 01/07/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Newman, Michael, Gillet, Anthony, Krowitz, David Mark, Edgington, Michael D.
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US13/345,219
• Publication Number:
  US 20120179457A1
• Time in Patent Office: 2,300 Days
• Field of Search: 704201, 704231, 704257, 704275, 704277
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques

Abstract

Techniques for combining the results of multiple recognizers in a distributed speech recognition architecture. Speech data input to a client device is encoded and processed both locally and remotely by different recognizers configured to be proficient at different speech recognition tasks. The client/server architecture is configurable to enable network providers to specify a policy directed to a trade-off between reducing recognition latency perceived by a user and usage of network resources. The results of the local and remote speech recognition engines are combined based, at least in part, on logic stored by one or more components of the client/server architecture.

Claims

1. A method of performing speech recognition in a distributed speech recognition system comprising an electronic device including an embedded speech recognizer and a network device including a remote speech recognizer remote from the electronic device, the method comprising:

receiving, by the electronic device, input audio comprising speech;

transmitting at least a portion of the input audio to the network device for processing by the remote speech recognizer to produce a remote speech recognition result;

processing, by the embedded speech recognizer, at least a portion of the input audio to produce a local speech recognition result;

identifying based on a command word included in the local speech recognition result, a mobile phone application;

performing a partial action on the electronic device, based, at least in part, on the local speech recognition result,

wherein performing a partial action comprises starting the identified mobile phone application on the electronic device, wherein performing the partial action is initiated prior to receiving the remote speech recognition result from the network device;

receiving, from the network device, the remote speech recognition result; and

performing a full action on the electronic device that completes the partial action based, at least in part, and responsive to receiving the remote speech recognition result from the network device.

2. The method of claim 1, wherein transmitting at least a portion of the input audio to the network device comprises transmitting a processed version of the at least a portion of the input audio to the network device.

3. The method of claim 2, wherein the processed version is a compressed version of the at least a portion of the input audio.

4. The method of claim 1, further comprising:

determining a confidence value associated with the local speech recognition result;

wherein performing the full action that completes the partial action comprises performing the full action in response to determining that the confidence value is greater than a predetermined threshold value.

5. The method of claim 1, further comprising:

determining a confidence value associated with the local speech recognition result; and

performing the partial action in response to determining that the confidence value is below a predetermined threshold value.

6. The method of claim 1, wherein the mobile phone application is configured to send a communication including text to another electronic device and wherein performing the partial action further comprises:

filling in at least one first field in the communication based, at least in part, on the local speech recognition result.

7. The method of claim 6, further comprising:

receiving from the network device a remote speech recognition result; and

filling in at least one second field in the communication based, at least in part, on the remote speech recognition result.

8. The method of claim 1, further comprising:

determining whether the full action to complete the partial action can be performed without receiving the remote speech recognition result from the network device;

wherein performing the full action that completes the partial action comprises performing the full action in response to determining that the full action can be performed without receiving the remote speech recognition result from the network device.

9. The method of claim 1, further comprising:

determining that the input audio does not include enough information to perform the full action to complete the partial action; and

prompting the user of the electronic device to provide additional information in response to determining that the input audio does not include enough information to perform the full action.

10. The method of claim 1, wherein starting the mobile phone application on the electronic device comprises starting an application selected from the group consisting of an text-messaging application, an email application, a phone call application, and a web-based application.

11. The method of claim 1, further comprising:

determining that a confidence value associated with the local speech recognition result is mixed indicating that a first portion of the local speech recognition result is recognized with a high confidence and a second portion of the local speech recognition result is recognized with a low confidence; and

wherein performing the partial action on the electronic device comprises performing the partial action based, at least in part, on the first portion of the local speech recognition result.

12. A non-transitory computer-readable storage medium encoded with a plurality of instructions that, when executed by at least one processor on an electronic device in a distributed speech recognition system comprising the electronic device having an embedded speech recognizer and a network device having a remote speech recognizer remote from the electronic device, perform a method comprising:

receiving, by the electronic device, input audio comprising speech;

transmitting at least a portion of the input audio to the network device for processing by the remote speech recognizer to produce a remote speech recognition result;

processing, by the embedded speech recognizer, at least a portion of the input audio to produce a local speech recognition result;

identifying, based on a command word included in the local speech recognition result, a mobile phone application;

performing a partial action on the electronic device, based, at least in part, on the local speech recognition result, wherein performing a partial action comprises starting the identified mobile phone application on the electronic device, wherein performing the partial action is initiated prior to receiving the remote speech recognition result from the network device;

receiving, from the network device, the remote speech recognition result; and

performing a full action on the electronic device that completes the partial action based, at least in part, and responsive to receiving the remote speech recognition result from the network device.

13. The computer-readable storage medium of claim 12, further comprising:

determining a confidence value associated with the local speech recognition result;

wherein performing the full action to complete the partial action comprises performing the full action in response to determining that the confidence value is greater than a predetermined threshold value.

14. The computer-readable storage medium of claim 12, wherein starting the mobile phone application on the electronic device comprises starting an application selected from the group consisting of a text-messaging application, an email application, a phone call application, and a web-based application.

15. An electronic device for use in a distributed speech recognition system comprising the electronic device and a network device remote from the electronic device, the electronic device, comprising:

at least one storage device configured to store one or more applications;

an embedded speech recognizer configured to;

receive input audio comprising speech;

transmit at least a portion of the input audio to the network device for processing by the remote speech recognizer to produce a remote speech recognition result;

process at least a portion of the input audio to produce a local speech recognition result;

identify, based on a command word included in the local speech recognition result, a partial action to be performed;

perform, in response to producing the local speech recognition result, the partial action on the electronic device, based, at least in part, on the local speech recognition result;

provide an indication of the performed partial action on the electronic device to enable a user of the electronic device to observe the results of the partial action prior to receiving the remote speech recognition result, wherein performing the partial action and providing the indication of the performed partial action are initiated prior to receiving the remote speech recognition result from the network device; and

perform a full action on the electronic device that completes the partial action based, at least in part, and responsive to receiving the remote speech recognition result from the network device.

16. The electronic device of claim 15, wherein the embedded speech recognizer is configured to:

receive from the network device the remote speech recognition result;

wherein performing the full action is based, at least in part, on the remote speech recognition result received from the network device.

17. The electronic device of claim 15, wherein performing the partial action comprises:

starting at least one of the one or more applications stored on the at least one storage device.