METHODS AND APPARATUS FOR HYBRID SPEECH RECOGNITION PROCESSING

• US 20180197545A1
• Filed: 01/11/2017
• Published: 07/12/2018
• Est. Priority Date: 01/11/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Willett, Daniel, Pinto, Joel, Ganong, III, William F.
• Granted Patent: US 10,971,157 B2
• Time in Patent Office: Days
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Methods and apparatus for selectively performing speech processing in a hybrid speech processing system. The hybrid speech processing system includes at least one mobile electronic device and a network-connected server remotely located from the at least one mobile electronic device. The mobile electronic device is configured to use an embedded speech recognizer to process at least a portion of input audio to produce recognized text. A controller on the mobile electronic device determines whether to send information from the mobile electronic device to the server for speech processing. The determination of whether to send the information is based, at least in part, on an analysis of the input audio, the recognized text, or a semantic category associated with the recognized text.

Claims

1. A mobile electronic device for use in a hybrid speech processing system comprising the mobile electronic device and a network-connected server remotely located from the mobile electronic device, the mobile electronic device comprising:

an input interface configured to receive input audio comprising speech;

an embedded speech recognizer configured to process at least a portion of the input audio to produce first recognized text;

a controller configured to determine whether to send information to the server for speech processing, wherein the information includes at least a portion of the input audio and/or at least a portion of the first recognized text, wherein the determination of whether to send the information is based, at least in part, on an analysis of the input audio, the recognized text, or a semantic category associated with the first recognized text; and

a network interface configured to send the information to the server in response to determining that the information should be sent to the server.

2. The mobile electronic device of claim 1, wherein the embedded speech recognizer is configured to perform speech recognition in a first language, and wherein determining whether to send information to the server for speech processing comprises:

determining that the speech includes at least one word in a second language different from the first language; and

determining to send at least a portion of the speech to the server when it is determined that the speech includes at least one word in the second language.

3. The mobile electronic device of claim 2, wherein the embedded speech recognizer is configured to perform speech recognition using at least one model that includes information about the first language and the second language;

and

wherein determining that the speech includes at least one word in the second language is based, at least in part, on an identification by the embedded speech recognizer that the speech includes the at least one word in the second language.

4. The mobile electronic device of claim 2, wherein the controller is further configured to apply a language identification process to at least a portion of the speech to identify a language of the speech;

and wherein

determining that the speech includes at least one word in the second language comprises detecting a mismatch between the language identified by the language identification process and the first language for which the embedded speech recognizer is configured to perform speech recognition.

5. The mobile electronic device of claim 4, wherein the embedded speech recognition engine is further configured to output a confidence score associated with the first recognized text;

and

wherein the controller is configured to apply the language identification process to at least a portion of the speech only when the confidence score is below a threshold value.

6. The mobile electronic device of claim 2, wherein the information sent to the server includes at least a portion of the input audio and an indication that the speech includes at least one word in the second language.

7. The mobile electronic device of claim 2, wherein the controller is further configured to:

receive second recognized text from the server, wherein the second recognized text corresponds to the at least a portion of the speech sent to the server;

combine at least a portion of the first recognized text corresponding to the first language and the second recognized text corresponding to the second language into a multi-language speech recognition result; and

output the multi-language speech recognition result.

8. The mobile electronic device of claim 1, further comprising:

an embedded natural language understanding (NLU) engine configured to process at least a portion of the recognized text to determine a semantic category associated with the recognized text;

wherein determining whether to send information to the server for speech processing comprises determining whether to send information to the server for speech processing based, at least in part, on the determined semantic category.

9. The mobile electronic device of claim 8, further comprising:

at least one storage device configured to store privacy settings for transmitting information to the server;

wherein the controller is further programmed to present a user interface on the mobile electronic device, wherein the user interface enables a user of the mobile electronic device to specify at least some of the privacy settings stored on the at least one storage device; and

wherein determining whether to send information to the server for speech processing further comprises determining whether to send information to the server for speech processing based, at least in part, on the stored privacy settings.

10. The mobile electronic device of claim 9, wherein the user interface enables the user to specify general privacy settings that are not category specific, and wherein the stored privacy settings are determined based, at least in part on the general privacy settings specified by the user.

11. The mobile electronic device of claim 9, wherein the user interface enables the user to specify privacy settings for each of a plurality of semantic categories.

12. The mobile electronic device of claim 1, wherein the information sent to the server includes at least a portion of the recognized text and does not include any of the input audio.

13. A method for use in a hybrid speech processing system comprising a mobile electronic device and a network-connected server remotely located from the mobile electronic device, the method comprising:

processing, by an embedded speech recognizer on the mobile electronic device, at least a portion of input audio to produce recognized text;

determining, by a controller, whether to send information from the mobile electronic device to the server for speech processing, wherein the information includes at least a portion of the input audio and/or at least a portion of the recognized text, wherein the determination of whether to send the information is based, at least in part, on an analysis of the input audio, the recognized text, or a semantic category associated with the recognized text; and

sending the information from the mobile electronic device to the server in response to determining that the information should be sent to the server.

14. The method of claim 13, wherein processing at least a portion of input audio to produce recognized text comprises performing speech recognition in a first language, and wherein determining whether to send information to the server for speech processing comprises:

determining that the speech includes at least one word in a second language different from the first language; and

determining to send at least a portion of the speech to the server when it is determined that the speech includes at least one word in the second language.

15. The method of claim 14, wherein performing speech recognition in a first language comprises using at least one model that includes information about the first language and the second language;

and

wherein determining that the speech includes at least one word in the second language is based, at least in part, on an identification by the embedded speech recognizer that the speech includes the at least one word in the second language.

16. The method of claim 13, further comprising:

processing, by an embedded natural language understanding (NLU) engine, at least a portion of the recognized text to determine a semantic category associated with the recognized text;

wherein determining whether to send information to the server for speech processing comprises determining whether to send information to the server for speech processing based, at least in part, on the determined semantic category.

17. A server for use in a hybrid speech processing system comprising the server and a network-connected mobile electronic device remotely located from the server, wherein the mobile electronic device comprises an embedded speech recognizer, the server comprising:

a controller configured to;

determine based, at least in part, on text received from the mobile electronic device, whether to request speech associated with the text from the mobile electronic device; and

request speech associated with the text received from the mobile device in response to determining to request the speech;

a server speech recognizer configured to process at least a portion of the speech received from the mobile electronic device to produce recognized text; and

a network interface configured to send the recognized text to the mobile electronic device.

18. The server of claim 17, further comprising:

a natural language understanding (NLU) engine configured to process at least a portion of the text received from the mobile electronic device to generate an NLU result; and

wherein determining whether to request speech associated with the text is further based, at least in part, on the NLU result.

19. The server of claim 18, further comprising:

at least one storage device configured to store semantic category information;

wherein the NLU result is a semantic category associated with the text; and

wherein determining whether to request speech associated with the text is further based, at least in part, on a comparison of the semantic category associated with the text and the stored semantic category information.

20. The server of claim 17, wherein determining whether to request speech associated with the text is further based, at least in part, on semantic category information received from the mobile electronic device.