Hybrid speech recognition

• US 8,249,877 B2
• Filed: 09/24/2010
• Issued: 08/21/2012
• Est. Priority Date: 08/29/2008
• Status: Active Grant

• Current Assignee: Solventum Intellectual Properties Company (Solventum Corp.)
• Original Assignee: Multimodal Technologies Incorporated (3M Company)
• Inventors: Koll, Detlef
• Primary Examiner(s): Chawan, Vijay B
• Application Number:
  US12/890,280
• Publication Number:
  US 20110238415A1
• Time in Patent Office: 697 Days
• Field of Search: 704/270, 704/270.1, 704/231, 704/275, 704/235, 704/256.5, 704/254, 704/251, 704/257, 704/243, 379/88.01, 379/88.02
• US Class Current: 704/270.1
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

A hybrid speech recognition system uses a client-side speech recognition engine and a server-side speech recognition engine to produce speech recognition results for the same speech. An arbitration engine produces speech recognition output based on one or both of the client-side and server-side speech recognition results.

Claims

1. A computer-implemented method performed by a client device, the method comprising:

(A) receiving a request from a requester to apply automatic speech recognition to an audio signal;

(B) providing the audio signal to a first automatic speech recognition engine in the client device;

(C) receiving first speech recognition results from the first automatic speech recognition engine;

(D) determining whether a second automatic speech recognition engine, in a server device, is accessible to the client device;

(E) if the second automatic speech recognition engine is determined not to be accessible to the client device, then providing the first speech recognition results to the requester in response to the request.

2. The method of claim 1, wherein the requester comprises a machine coupled to the client device.

3. The method of claim 1, wherein the requester comprises software executing on the client device.

4. An apparatus comprising:

means for receiving a request from a requester to apply automatic speech recognition to an audio signal;

means for providing the audio signal to a first automatic speech recognition engine in the client device;

means for receiving first speech recognition results from the first automatic speech recognition engine;

means for determining whether a second automatic speech recognition engine, in a server device, is accessible to the client device; and

means for providing the first speech recognition results to the requester in response to the request if the second automatic speech recognition engine is determined not to be accessible to the client device.

5. A computer-implemented method performed by a client device, the method comprising:

(A) receiving a request from a requester to apply automatic speech recognition to an audio signal;

(B) providing the audio signal to a first automatic speech recognition engine in a server device;

(C) receiving first speech recognition results from the first automatic speech recognition engine;

(D) determining whether a second automatic speech recognition engine, in the client device, is accessible to the client device;

(E) if the second automatic speech recognition engine is determined not to be accessible to the client device, then providing the first speech recognition results to the requester in response to the request.

6. An apparatus comprising:

means for receiving a request from a requester to apply automatic speech recognition to an audio signal;

means for providing the audio signal to a first automatic speech recognition engine in a server device;

means for receiving first speech recognition results from the first automatic speech recognition engine;

means for determining whether a second automatic speech recognition engine, in the client device, is accessible to the client device;

means for providing the first speech recognition results to the requester in response to the request if the second automatic speech recognition engine is determined not to be accessible to the client device.

7. A computer-implemented method performed by a client device, the method comprising:

(A) receiving a request from a requester to apply automatic speech recognition to an audio signal;

(B) providing the audio signal to a first automatic speech recognition engine in the client device;

(C) providing the audio signal to a second automatic speech recognition engine in a server device;

(D) receiving first speech recognition results from the first automatic speech recognition engine;

(E) determining whether a confidence measure associated with the first speech recognition results exceeds a predetermined threshold; and

(F) if the confidence measure exceeds the predetermined threshold, then providing the first speech recognition results to the requester in response to the request.

8. The method of claim 7, further comprising:

(G) before (F), receiving second speech recognition results from the second automatic speech recognition engine; and

wherein (F) comprises providing the first speech recognition results but not the second speech recognition results to the requester.

9. An apparatus comprising:

means for receiving a request from a requester to apply automatic speech recognition to an audio signal;

means for providing the audio signal to a first automatic speech recognition engine in the client device;

means for providing the audio signal to a second automatic speech recognition engine in a server device;

means for receiving first speech recognition results from the first automatic speech recognition engine;

means for determining whether a confidence measure associated with the first speech recognition results exceeds a predetermined threshold; and

means for providing the first speech recognition results to the requester in response to the request if the confidence measure exceeds the predetermined threshold.

10. A computer-implemented method performed by a client device, the method comprising:

(A) receiving a request from a requester to apply automatic speech recognition to an audio signal;

(B) providing the audio signal to a first automatic speech recognition engine in the client device;

(C) receiving first speech recognition results from the first automatic speech recognition engine;

(D) providing the audio signal to a second automatic speech recognition engine in a server device;

(E) receiving second speech recognition results from the second automatic speech recognition engine;

(F) producing hybrid speech recognition results based on the first speech recognition results and the second speech recognition results; and

(G) providing the hybrid speech recognition results to the requester in response to the request.

11. The method of claim 10, wherein (F) comprises combining the first speech recognition results and the second speech recognition results using Recognizer Output Voting Error Reduction.

12. The method of claim 10, wherein the client device is configured to treat one of the first and second automatic speech recognition engines as a preferred speech recognition engine, and:

wherein (C) comprises receiving the first speech recognition results at an arbitration engine in the client device at a first time;

wherein (E) comprises receiving the second speech recognition results at the arbitration engine in the client device at a second time that is later than the first time; and

wherein (F) comprises;

(F)(1) including the first recognition results in the hybrid speech recognition results; and

(F)(2) including the second speech recognition results in the hybrid speech recognition results only if the second automatic speech recognition engine is the preferred speech recognition engine.

13. The method of claim 10, wherein the client device is configured to treat one of the first and second automatic speech recognition engines as a preferred speech recognition engine, and:

wherein (E) comprises receiving the second speech recognition results at an arbitration engine in the client device at a first time;

wherein (C) comprises receiving the first speech recognition results at the arbitration engine in the client device at a second time that is later than the first time; and

wherein (F) comprises;

(F)(1) including the second recognition results in the hybrid speech recognition results; and

(F)(2) including the first speech recognition results in the hybrid speech recognition results only if the first automatic speech recognition engine is the preferred speech recognition engine.

14. The method of claim 10:

wherein (C) comprises receiving the first speech recognition results at an arbitration engine in the client device at a first time;

wherein (E) comprises receiving the second speech recognition results at the arbitration engine in the client device at a second time that is later than the first time; and

wherein (F) comprises;

(F)(1) including the first recognition results in the hybrid speech recognition results;

(F)(2) identifying words in the second speech recognition results that do not overlap in time with any words in the first speech recognition results; and

(F)(3) including only non-overlapping words from the second speech recognition results in the hybrid speech recognition results.

15. The method of claim 10:

wherein (E) comprises receiving the second speech recognition results at an arbitration engine in the client device at a first time;

wherein (C) comprises receiving the first speech recognition results at the arbitration engine in the client device at a second time that is later than the first time; and

wherein (F) comprises;

(F)(1) including the second recognition results in the hybrid speech recognition results;

(F)(2) identifying words in the first speech recognition results that do not overlap in time with any words in the second speech recognition results; and

(F)(3) including only non-overlapping words from the first speech recognition results in the hybrid speech recognition results.

16. The method of claim 10, wherein (F) comprises:

(F)(1) including the first recognition results in the hybrid speech recognition results; and

(F)(2) replacing the first recognition results with the second recognition results in the hybrid speech recognition results.

17. The method of claim 10, wherein (F) comprises:

(F)(1) including the second recognition results in the hybrid speech recognition results; and

(F)(2) replacing the second recognition results with the first recognition results in the hybrid speech recognition results.

18. An apparatus comprising:

means for receiving a request from a requester to apply automatic speech recognition to an audio signal;

means for providing the audio signal to a first automatic speech recognition engine in the client device;

means for receiving first speech recognition results from the first automatic speech recognition engine;

means for providing the audio signal to a second automatic speech recognition engine in a server device;

means for receiving second speech recognition results from the second automatic speech recognition engine;

means for producing hybrid speech recognition results based on the first speech recognition results and the second speech recognition results; and

means for providing the hybrid speech recognition results to the requester in response to the request.