Hybridized client-server speech recognition

• US 9,674,328 B2
• Filed: 02/22/2012
• Issued: 06/06/2017
• Est. Priority Date: 02/22/2011
• Status: Active Grant

• Current Assignee: Speak With Me, Inc.
• Original Assignee: Speak With Me, Inc.
• Inventors: Juneja, Ajay
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US13/402,678
• Publication Number:
  US 20120215539A1
• Time in Patent Office: 1,931 Days
• Field of Search: 704 8, 704231, 704233, 704236, 7042701
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34
• CPC Class Codes: G10L 25/60 for measuring the quality of voice signals
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/0894 Packet rate
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/63 Routing a service request depending on the request content or context
• CPC Class Codes: H04M 1/271 controlled by voice recognition
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72436 for text messaging, e.g. short messaging services [SMS] or e-mails
• CPC Class Codes: H04M 2250/14 including a card reading device
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A recipient computing device can receive a speech utterance to be processed by speech recognition and segment the speech utterance into two or more speech utterance segments, each of which can be to one of a plurality of available speech recognizers. A first one of the plurality of available speech recognizers can be implemented on a separate computing device accessible via a data network. A first segment can be processed by the first recognizer and the results of the processing returned to the recipient computing device, and a second segment can be processed by a second recognizer implemented at the recipient computing device.

Claims

1. A computer program product comprising a non-transitory computer-readable storage medium storing instructions that, when executed by a computing system comprising at least one programmable processor, cause the computing system to perform operations comprising:

receiving, at a recipient computing device, a speech utterance to be processed by speech recognition;

determining an amount of an available bandwidth between the recipient computing device and a separate computing device;

segmenting, upon determining that the available bandwidth is sufficient, the speech utterance into two or more speech utterance segments, the segmenting comprising initially analyzing the speech utterance by identifying features of the speech utterance that can be more efficiently processed by the separate computing device than the recipient computing device, wherein initially analyzing comprises applying a dynamically adaptable acoustic model implemented at the recipient computing device, with the dynamically adaptable acoustic model adjusted based on locally available data at the recipient computing device including a user location and time, to determine a confidence score, and an audio quality metric for the two or more speech utterance segments;

dynamically determining a confidence threshold value and an audio quality threshold value based on environmental conditions at which the recipient computing device is located, the environmental conditions comprising one or more of;

a type of environment in which the recipient computing device is located, availability of noise cancelling devices at the recipient computing device, and number of microphones used by the recipient computing device;

assigning each of the two or more speech utterance segments to one or more of a plurality of available speech recognizers, the assigning comprising;

designating a first segment of the two or more speech utterance segments for processing by a first speech recognizer of the plurality of available speech recognizers that is implemented on the separate computing device than the recipient computing device, wherein designating the first segment is performed when at least one of the confidence score and the audio quality metric for the first segment, determined using the dynamically adaptable acoustic model adjusted based on the locally available data including the user location and the time, are below the respective confidence threshold value and the audio quality threshold value, and

designating a second segment of the two or more speech utterance segments for processing by a second speech recognizer of the plurality of available speech recognizers that is implemented on the recipient computing device when another confidence score and another audio quality metric for the second segment, determined using the dynamically adaptable acoustic model adjusted based on the locally available data including the user location and the time, are above the respective confidence threshold value and the audio quality threshold value,

wherein the identifying of the features of the speech utterance comprising determining processing speeds associated with the separate computing device and the recipient computing device, the available bandwidth, and a presence of a word or phrase capable of being efficiently modeled by a context-free grammar at the recipient computing device;

sending the first segment from the recipient computing device to the separate computing device for processing;

receiving first segment processing results back from the separate computing device, the sending and the receiving occurring via a data network;

processing the second segment at the recipient computing device to generate second segment processing results; and

returning a completed speech recognition result assembled from the first segment processing results and the second segment processing results.

2. A computer program product as in claim 1, wherein the identifying of the features of the speech utterance further comprises analyzing the speech utterance using the dynamically adaptable acoustic model implemented on one or more processors at the recipient computing device.

3. A computer program product as in claim 1, wherein the respective threshold values are determined based on one or more criteria defined at design time or dynamically evaluated at run time.

4. A computer program product as in claim 1, wherein the recipient computing device comprises a thin client computing device or terminal and the separate computing device comprises at least one server accessible over the data network from the thin client computing device or terminal.

5. A computer program product as in claim 1, wherein the assigning further comprises designating the first segment for processing by the first speech recognizer implemented on the separate computing device when the first segment is determined by the initially analyzing to include one or more words that relate to data that are more readily accessible at the separate computing device than at the recipient computing device.

6. A computer program product as in claim 5, wherein the first segment processing results comprise analysis of the first segment performed by the separate computing device using an adapted language model, the adapted language model being adapted based on the data that are more readily accessible at the separate computing device than at the recipient computing device.

7. A recipient computing device comprising:

a computing system comprising at least one programmable processor;

a computer-readable storage medium storing instructions that, when executed by the computing system, cause the at least one programmable processor of the computing system to perform operations comprising;

determining an amount of an available bandwidth between the recipient computing device and a separate computing device;

segmenting, upon determining that the available bandwidth is sufficient, the speech utterance into two or more speech utterance segments, the segmenting comprising initially analyzing the speech utterance by identifying features of the speech utterance that can be more efficiently processed by the separate computing device than the recipient computing device, wherein initially analyzing comprises applying a dynamically adaptable acoustic model implemented at the recipient computing device, with the dynamically adaptable acoustic model adjusted based on locally available data at the recipient computing device including a user location and time, to determine a confidence score and an audio quality metric for the two or more speech utterance segments;

dynamically determining a confidence threshold value and an audio quality threshold value based on environmental conditions at which the recipient computing device is located, the environmental conditions comprising one or more of;

a type of environment in which the recipient computing device is located, availability of noise cancelling devices at the recipient computing device, and number of microphones used by the recipient computing device;

assigning each of the two or more speech utterance segments to one or more of a plurality of available speech recognizers, the assigning comprising;

designating a first segment of the two or more speech utterance segments for processing by a first speech recognizer of the plurality of available speech recognizers that is implemented on the separate computing device than the recipient computing device, wherein designating the first segment is performed when at least one of the confidence score and the audio quality metric for the first segment, determined using the dynamically adaptable acoustic model adjusted based on the locally available data including the user location and the time, are below the respective confidence threshold value and the audio quality threshold value, and

designating a second segment of the two or more speech utterance segments for processing by a second speech recognizer of the plurality of available speech recognizers that is implemented on the recipient computing device when another confidence score and another audio quality metric for the second segment, determined using the dynamically adaptable acoustic model adjusted based on the locally available data including the user location and the time, are above the respective confidence threshold value and the audio quality threshold value,

wherein the identifying of the features of the speech utterance comprising determining processing speeds associated with the separate computing device and the recipient computing device, the available bandwidth, and a presence of a word or phrase capable of being efficiently modeled by a context-free grammar at the recipient computing device;

sending the first segment from the recipient computing device to the separate computing device for processing;

receiving first segment processing results back from the separate computing device, the sending and the receiving occurring via a data network;

processing the second segment at the recipient computing device to generate second segment processing results; and

returning a completed speech recognition result assembled from the first segment processing results and the second segment processing results.

8. A recipient computing device as in claim 7, wherein the identifying of the features of the speech utterance further comprises analyzing the speech utterance using the dynamically adaptable acoustic model implemented on one or more processors at the recipient computing device.

9. A recipient computing device as in claim 7, wherein the respective threshold values are determined based on one or more criteria defined at design time or dynamically evaluated at run time.

10. A recipient computing device as in claim 7, wherein the recipient computing device comprises a thin client computing device or terminal and the separate computing device comprises at least one server accessible over the data network from the thin client computing device or terminal.

11. A recipient computing device as in claim 7, wherein the assigning further comprises designating the first segment for processing by the first speech recognizer implemented on the separate computing device when the first segment is determined by the initially analyzing to include one or more words that relate to data that are more readily accessible at the separate computing device than at the recipient computing device.

12. A recipient computing device as in claim 11, wherein the first segment processing results comprise analysis of the first segment performed by the separate computing device using an adapted language model, the adapted language model being adapted based on the data that are more readily accessible at the separate computing device than at the recipient computing device.

13. A method comprising:

receiving, at a recipient computing device comprising a computing system comprising at least one programmable processor, a speech utterance to be processed by speech recognition;

determining an amount of an available bandwidth between the recipient computing device and a separate computing device;

segmenting, upon determining that the available bandwidth is sufficient, the speech utterance into two or more speech utterance segments, the segmenting comprising initially analyzing the speech utterance by identifying features of the speech utterance that can be more efficiently processed by the separate computing device than the recipient computing device, wherein initially analyzing comprises applying a dynamically adaptable acoustic model implemented at the recipient computing device, with the dynamically adaptable acoustic model adjusted based on locally available data at the recipient computing device including a user location and time, to determine a confidence score and an audio quality metric for the two or more speech utterance segments;

dynamically determining a confidence threshold value and an audio quality threshold value based on environmental conditions at which the recipient computing device is located, the environmental conditions comprising one or more of;

a type of environment in which the recipient computing device is located, availability of noise cancelling devices at the recipient computing device, and number of microphones used by the recipient computing device;

assigning each of the two or more speech utterance segments to one or more of a plurality of available speech recognizers, the assigning comprising;

designating a first segment of the two or more speech utterance segments for processing by a first speech recognizer of the plurality of available speech recognizers that is implemented on the separate computing device than the recipient computing device, wherein designating the first segment is performed when at least one of the confidence score and the audio quality metric for the first segment, determined using the dynamically adaptable acoustic model adjusted based on the locally available data including the user location and the time, are below the respective confidence threshold value and the audio quality threshold value, and

designating a second segment of the two or more speech utterance segments for processing by a second speech recognizer of the plurality of available speech recognizers that is implemented on the recipient computing device when another confidence score and another audio quality metric for the second segment, determined using the dynamically adaptable acoustic model adjusted based on the locally available data including the user location and the time, are above the respective confidence threshold value and the audio quality threshold value,

wherein the identifying of the features of the speech utterance comprising determining processing speeds associated with the separate computing device and the recipient computing device, the available bandwidth, and a presence of a word or phrase capable of being efficiently modeled by a context-free grammar at the recipient computing device;

sending, by the computing system, the first segment from the recipient computing device to the separate computing device for processing;

receiving, by the computing system, first segment processing results back from the separate computing device, the sending and the receiving occurring via a data network;

processing, by the computing system, the second segment at the recipient computing device to generate second segment processing results; and

returning, by the computing system, a completed speech recognition result assembled from the first segment processing results and the second segment processing results.

14. A method as in claim 13, wherein the identifying of the features of the speech utterance further comprises analyzing the speech utterance using the dynamically adaptable acoustic model implemented on one or more processors at the recipient computing device.

15. A method as in claim 13, wherein the respective threshold values are determined based on one or more criteria defined at design time or dynamically evaluated at run time.

16. A method as in claim 13, wherein the recipient computing device comprises a thin client computing device or terminal and the separate computing device comprises at least one server accessible over the data network from the thin client computing device or terminal.

17. A method as in claim 13, wherein the assigning further comprises designating the first segment for processing by the first speech recognizer implemented on the separate computing device when the first segment is determined by the initially analyzing to include one or more words that relate to data that are more readily accessible at the separate computing device than at the recipient computing device.

18. A method as in claim 13, wherein initially analyzing the speech utterance by identifying the features of the speech utterance that can be more efficiently processed by the separate computing device than the recipient computing device comprises:

identifying a feature of the speech utterance as commands-based speech data corresponding to the second segment to be processed at the second speech recognizer implemented at the recipient computing device; and

identifying another feature of the speech utterance as additional information, including one or more of dictation data, music or video data, user acoustic profile data, or foreign language speech data, the additional information being related to the identified command-based speech data corresponding to the second segment, with the other feature of the speech utterance to be processed at the first speech recognizer implemented on the separate computing device.

19. A method as in claim 13, wherein sending the first segment from the recipient computing device to the separate computing device for processing comprises:

sending the first segment from the recipient computing device to the separate computing device for processing using an adapted language model that is one or more of;

time sensitive, and location sensitive.