Hybrid controller for ASR

• US 9,886,944 B2
• Filed: 10/04/2012
• Issued: 02/06/2018
• Est. Priority Date: 10/04/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Willett, Daniel, Wu, Jianxiong, Vozila, Paul J., Ganong, III, William F.
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US14/433,263
• Publication Number:
  US 20150279352A1
• Time in Patent Office: 1,951 Days
• Field of Search: 704231
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 19/0017 Lossless audio signal coding; Perfect reconstruction of coded audio signal by transmission of coding error G10L19/24 takes precedence

Abstract

A mobile device is described which is adapted for automatic speech recognition (ASR). A speech input receives an unknown speech input signal from a user. A local controller determines if a remote ASR processing condition is met, transforms the speech input signal into a selected one of multiple different speech representation types, and sends the transformed speech input signal to a remote server for remote ASR processing. A local ASR arrangement performs local ASR processing of the speech input including processing any speech recognition results received from the remote server.

Claims

1. A mobile device adapted for automatic speech recognition (ASR) of speech input, the mobile device comprising:

a local ASR arrangement configured to perform local ASR processing of the speech input including processing any speech recognition results received from a remote server; and

a local controller configured to perform;

sending the speech input to the local ASR arrangement;

sending the speech input to the remote server for remote ASR processing; and

after sending the speech input to the remote server, determining if a remote ASR processing condition is met,

wherein the local controller determines if the remote ASR processing condition is met while the local ASR arrangement processes the speech input, and

wherein local ASR recognition processes are suspended by the local ASR arrangement in response to the local controller determining that the remote ASR processing condition is met so that recognition results solely from the remote ASR processing are used to produce an output interpretation of the speech input.

2. The mobile device according to claim 1, wherein the local controller transforms the speech input into a selected one of a plurality of different speech representation types, and wherein the local controller transforms the speech input signal and sends the transformed speech input.

3. The mobile device according to claim 2, wherein the local controller suspends transforming the speech input signal and sending the transformed speech input signal if it determines that the remote ASR condition is not met.

4. The mobile device according to claim 1, wherein the local controller transforms the speech input signal into a selected one of a plurality of different speech representation types, and wherein the local controller selects one of the speech representation types based on different bandwidth characteristics of the speech representation types.

5. The mobile device according to claim 1, wherein the remote ASR processing condition is a function of a connection condition between the mobile device and the remote server.

6. The mobile device according to claim 1, wherein the remote ASR processing condition is a function of a projected accuracy benefit associated with the remote ASR processing.

7. The mobile device according to claim 6, wherein the projected accuracy benefit is based on a metadata function reflecting one or both of application state and dialog context.

8. The mobile device according to claim 1, wherein the remote ASR processing condition is a function of one or both of a local ASR processing latency characteristic and a remote ASR processing latency characteristic.

9. The mobile device according to claim 1, wherein the remote ASR processing condition is a function of a recognition cost characteristic.

10. The mobile device according to claim 1, wherein the local controller transforms the speech input signal into a selected one of a plurality of different speech representation types, and wherein the plurality of different speech recognition types include one or more of ASR feature vectors, lossy compressed speech, lossless compressed speech, and uncompressed speech.

11. The mobile device according to claim 1, wherein determining if the remote ASR processing condition is met reflects an adaptation process based on speech recognition operation over time.

12. The mobile device according to claim 11, wherein the adaptation process is an automatic unsupervised adaptation process.

13. A method for automatic speech recognition (ASR) of a speech input, the method comprising:

sending the speech input to a local ASR arrangement;

sending the speech input to a remote server for remote ASR processing;

and after sending the speech input to the remote server, determining if a remote ASR processing condition is met, wherein determining if the remote ASR processing condition is met occurs while the local ASR arrangement processes the speech input;

and when the remote ASR processing condition is determined to be met, suspending speech processing by the local ASR arrangement so that recognition results solely from the remote ASR processing are used to produce an output interpretation of the speech input.

14. The method of claim 13, further comprising transforming the speech input into a selected one of a plurality of different speech representation types, and wherein sending the speech input comprises sending the transformed speech input.

15. The method of claim 13, further comprising transforming the speech input into a selected one of a plurality of different speech representation types based on different bandwidth characteristics of the speech representation types.

16. At least one non-transitory computer-readable storage medium storing processor-executable instructions that when executed by a computer system, cause the computer system to perform a method for automatic speech recognition (ASR) of a speech input, the method comprising:

sending the speech input to a local ASR arrangement;

sending the speech input to a remote server for remote ASR processing;

and after sending the speech input to the remote server, determining if a remote ASR processing condition is met, wherein determining if the remote ASR processing condition is met occurs while the local ASR arrangement processes the speech input;

and when the remote ASR processing condition is determined to be met, suspending speech processing by the local ASR arrangement so that recognition results solely from the remote ASR processing are used to produce an output interpretation of the speech input.

17. The at least one non-transitory computer-readable storage medium of claim 16, wherein the method further comprises transforming the speech input into a selected one of a plurality of different speech representation types, and wherein sending the speech input comprises sending the transformed speech input.