Method and apparatus for evaluating the accuracy of a speech recognition system

• US 6,507,816 B2
• Filed: 05/04/1999
• Issued: 01/14/2003
• Est. Priority Date: 05/04/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ortega, Kerry A.
• Primary Examiner(s): CHAWAN, VIJAY B
• Application Number:
  US09/304,812
• Publication Number:
  US 20020177999A1
• Time in Patent Office: 1,351 Days
• Field of Search: 704/235, 704/254, 704/270, 704/275, 704/231, 704/255, 704/244, 704/251, 704/252, 704/243, 704/256, 704/257, 704/241, 704/272, 704/250, 707/500, 707/501
• US Class Current: 704/235
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

A method and system for evaluating the accuracy of a computer speech recognition system counts and indexes the total number of words dictated and the number of words corrected. The corrections are tallied after being made in a correction window and include words contained in an alternative list as well as words input by the user and within a stored word database. A processor calculates the approximate accuracy of the speech recognition system as the ratio of the number of correct words to the total number of words dictated. An accuracy ratio is calculated for each dictation session and an overall ratio is calculated for all sessions combined. The system also keeps individual and overall indexes of the number of times the corrected words were in alternate lists or not within the word database and uses these indexes to calculate additional accuracy values.

Claims

1. In a computer system for speech recognition, a method for evaluating how accurately dictated words are recognized during a single dictation session by a speech recognition system, comprising the steps of:

counting said dictated words to create a total word index;

counting mis-recognized words to create a correction index;

calculating an accuracy value of said speech recognition system for said single dictation session according to said total word and correction indexes.

2. The method according to claim 1, wherein said accuracy value is calculated by subtracting said correction index from said total word index to create a recognition index and computing a ratio of said recognition index to said total word index.

3. The method according to claim 2, wherein said total word and recognition indexes are summed for multiple dictation sessions and said accuracy value is a ratio of said summed recognition index to said summed total word index.

4. The method according to claim 2, further comprising the steps of:

comparing each said mis-recognized word to at least one alternate word;

counting each time said mis-recognized words are corrected by said at least one alternate words so as to create an alternate index; and

updating said total word index according to said at least one alternate word.

5. The method according to claim 4, wherein said accuracy value is calculated by adding said alternate index to said recognition index to create an alternate recognition index and computing a ratio of said alternate recognition index to said total word index.

6. The method according to claim 5, wherein said total word and alternate recognition indexes are summed for multiple dictation sessions and said accuracy value is a ratio of said summed alternate recognition index to said summed total word index.

7. The method according to claim 5, further comprising the step of receiving corrected words from user input, wherein said total word index is updated according to said corrected words.

8. The method according to claim 7, further comprising the steps of:

searching a word database for each said user corrected word; and

counting each time said user corrected word is not within said word database so as to create an out-of-vocabulary index.

9. The method according to claim 8, wherein said accuracy value is calculated by adding said out-of-vocabulary index to said alternate recognition index to create a vocabulary recognition index and computing a ratio of said vocabulary recognition index to said total word index.

10. The method according to claim 9, wherein said total word and vocabulary indexes are summed for multiple dictation sessions and said accuracy value is a ratio of said summed vocabulary recognition index to said summed total word index.

11. The method of claim 1, further comprising:

executing a problem solving application if said accuracy value is less than a predetermined minimum accuracy value.

12. A system for evaluating the accuracy in which dictated words are recognized during a single dictation session by a computer speech recognition system, comprising:

counting means for determining the number of said dictated words so as to create a total word index and for determining the number of mis-recognized words so as to create a correction index;

calculating means for calculating an accuracy value of said speech recognition system for said single dictation session according to said total word and correction indexes.

13. The system as recited in claim 12, wherein said calculating means calculates said accuracy value by subtracting said correction index from said total word index to create a recognition index and computing a ratio of said recognition index to said total word index.

14. The system as recited in claim 13, further comprising a comparing means and an updating means, wherein:

said comparing means compares each said mis-recognized word to at least one alternate word;

said counting means counts each time said mis-recognized words are corrected by said alternate words so as to create an alternate index; and

said updating means updates said total word index according to said alternate words.

15. The system as recited in claim 14, wherein said calculating means calculates said accuracy value by adding said alternate index to said recognition index to create an alternate recognition index and computing a ratio of said alternate recognition index to said total word index.

16. The system as recited in claim 14, further comprising an input means for receiving corrected words from a user, wherein said total word index is updated according to said user corrected words.

17. The system as recited in claim 16, further comprising a searching means, wherein:

said searching means searches a word database for each said user corrected word; and

said counting means counts each time said user corrected word is not within said word database so as to create an out-of-vocabulary index.

18. The system as recited in claim 17, wherein said calculation means calculates said accuracy value by adding said out-of-vocabulary index to said alternate recognition index to create a vocabulary recognition index and computing a ratio of said vocabulary recognition index to said total word index.

19. A machine readable storage, having stored thereon a computer program having a plurality of code sections executable by a machine for causing the machine to perform the steps of:

counting dictated words to create a total word index;

counting mis-recognized words to create a correction index;

during a dictation session in a single speech recognition system, calculating an accuracy value of said speech recognition system for said single dictation session according to said total word and correction indexes.

20. The machine-readable storage of claim 19, further comprising:

executing a problem solving application if said accuracy value is less than a predetermined minimum accuracy value.