Determining speech recognition accuracy

• US 7,181,392 B2
• Filed: 07/16/2002
• Issued: 02/20/2007
• Est. Priority Date: 07/16/2002
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Moore, Victor S., Jaiswal, Peeyush, Gandhi, Shailesh B., Toon, Gregory L.
• Primary Examiner(s): Hudspeth; David
• Assistant Examiner(s): Sked; Matthew J.
• Application Number:
  US10/196,014
• Publication Number:
  US 20040015350A1
• Time in Patent Office: 1,680 Days
• Field of Search: None
• US Class Current: 704/235
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

A method of determining the accuracy of a speech recognition system can include identifying from a log of the speech recognition system a text result and attributes associated with the text result. An audio representation from which the text result was derived can be accessed. The audio representation can be processed with a reference speech recognition engine to determine a second text result. The text result can be compared with the second text result to determine an accuracy of the speech recognition system.

Claims

1. A method of determining the accuracy of a speech recognition system comprising:

identifying from a log of said speech recognition system a text result and attributes associated with said text result, wherein said log comprises a plurality of log entries, each log entry correlated with a different electronically stored audio segment defining an audio representation corresponding to a portion of text for which a text result and at least one attribute is provided, and wherein each attribute characterizes at least one of a configuration of the speech recognition system, a type of audio channel over which an audio segment is received, and whether the audio segment is received in response to a user prompt;

accessing an audio representation from which said text result was derived;

processing said audio representation with a reference speech recognition engine to determine a second text result; and

comparing said text result with said second text result to determine an accuracy of said speech recognition system.

2. The method of claim 1, further comprising:

repeating each step of claim 1 for additional text results of said speech recognition system to determine an accuracy statistic of said speech recognition system.

3. The method of claim 2, further comprising:

identifying at least one error condition specified in said log; and

determining said accuracy statistic for said at least one of said identified error conditions.

4. The method of claim 2, wherein said attributes specify timing information for said text result, said accessing an audio representation step further comprising:

identifying said audio representation from a plurality of audio representations according to said attributes of said text result.

5. The method of claim 2, further comprising:

manually determining text from said audio representation; and

comparing said first text result and said second text result with said manually determined text.

6. The method of claim 5, further comprising:

receiving audio properties of said audio representation.

7. The method of claim 6, further comprising:

adjusting the configuration of said reference speech recognition engine according to said audio properties of said audio representation.

8. The method of claim 6, further comprising:

altering acoustic models of said reference speech recognition engine according to said audio properties of said audio representation.

9. The method of claim 6, wherein said accuracy statistic is selected from the group consisting of a ratio of failed recognitions to total recognitions minus failed recognitions due to uncontrollable environmental elements, and a ratio of failed recognitions due to uncontrollable environmental elements to total recognitions.

10. The method of claim 6, wherein said accuracy statistic is selected from the group consisting of a total number of occurrences of unique words in which there was an attempt at recognition, a number of said occurrences of unique words which were successfully recognized, a number of said occurrences of unique words which were unsuccessfully recognized, and a number of failed attempts for said occurrences of unique words due to uncontrollable environmental elements.

11. The method of claim 2, wherein said accuracy statistic is selected from the group consisting of a ratio of the successful recognitions to total recognitions, a ratio of successful recognitions to total recognitions minus failed recognitions, and a ratio of failed recognitions to total recognitions.

12. A machine-readable storage, having stored thereon a computer program having a plurality of code sections executable by a machine for causing the machine to perform the steps of:

identifying from a log of said speech recognition system a text result and attributes associated with said text result, wherein said log comprises a plurality of log entries, each log entry correlated with a different electronically stored audio segment defining an audio representation corresponding to a portion of text for which a text result and at least one attribute is provided, and wherein each attribute characterizes at least one of a configuration of the speech recognition system, a type of audio channel over which an audio segment is received, and whether the audio segment is received in response to a user prompt;

accessing an audio representation from which said text result was derived;

processing said audio representation with a reference speech recognition engine to determine a second text result; and

comparing said text result with said second text result to determine an accuracy of said speech recognition system.

13. The machine-readable storage of claim 12, further comprising:

repeating each step of claim 12 for additional text results of said speech recognition system to determine an accuracy statistic of said speech recognition system.

14. The machine-readable storage of claim 13, further comprising:

identifying at least one error condition specified in said log; and

determining said accuracy statistic for said at least one of said identified error conditions.

15. The machine-readable storage of claim 13, wherein said attributes specify timing information for said text result, said accessing an audio representation step further comprising:

identifying said audio representation from a plurality of audio representations according to said attributes of said text result.

16. The machine-readable storage of claim 13, further comprising:

manually determining text from said audio representation; and

comparing said first text result and said second text result with said manually determined text.

17. The machine-readable storage of claim 16, further comprising:

receiving audio properties of said audio representation.

18. The machine-readable storage of claim 17, further comprising:

adjusting the configuration of said reference speech recognition engine according to said audio properties of said audio representation.

19. The machine-readable storage of claim 17, further comprising:

altering acoustic models of said reference speech recognition engine according to said audio properties of said audio representation.

20. The machine-readable storage of claim 17, wherein said accuracy statistic is selected from the group consisting of a ratio of failed recognitions to total recognitions minus failed recognitions due to uncontrollable environmental elements, and a ratio of failed recognitions due to uncontrollable environmental elements to total recognitions.

21. The machine-readable storage of claim 17, wherein said accuracy statistic is selected from the group consisting of a total number of occurrences of unique words in which there was an attempt at recognition, a number of said occurrences of unique words which were successfully recognized, a number of said occurrences of unique words which were unsuccessfully recognized, and a number of failed attempts for said occurrences of unique words due to uncontrollable environmental elements.

22. The machine-readable storage of claim 13, wherein said accuracy statistic is selected from the group consisting of a ratio of the successful recognitions to total recognitions, a ratio of successful recognitions to total recognitions minus failed recognitions, and a ratio of failed recognitions to total recognitions.