Performance gauge for a distributed speech recognition system

• US 6,766,294 B2
• Filed: 11/30/2001
• Issued: 07/20/2004
• Est. Priority Date: 11/30/2001
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: Dictaphone Corporation (Microsoft Corporation)
• Inventors: Greene, Channell, Cyr, James, Hold, Martin, Kuhnen, Regina, MacGinite, Andrew
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): HARPER, V PAUL
• Application Number:
  US09/997,243
• Publication Number:
  US 20030105630A1
• Time in Patent Office: 963 Days
• Field of Search: 707/536, 705/3, 705/2, 704/200, 704/231, 704/235, 704/243, 704/244, 704/251, 704/260, 704/270, 704/275
• US Class Current: 704/235
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems

Abstract

A performance gauge for use in conjunction with a transcription system including a speech processor linked to at least one speech recognition engine and at least one transcriptionist. The speech processor includes an input for receiving speech files and storage means for storing the received speech files until such a time that they are forwarded to a selected speech recognition engine or transcriptionist for processing. The system includes a transcriptionist text file database in which manually transcribed transcriptionist text files are stored, each stored transcriptionist text file including time stamped data indicative of position within an original speech file. The system further includes a recognition engine text file database in which recognition engine text files transcribed via the at least one speech recognition engine are stored, each stored recognition engine text file including time stamped data indicative of position within an original speech file. The system further includes a comparator comparing the time stamped recognition engine text files with time stamped transcriptionist text files based upon the same speech file so as to determine differences between the recognition engine text file and the transcriptionist text file and a report generator compiling the identified differences so as to issue a report for evaluation by administrators of the transcription system.

Claims

1. A performance gauge configured to interface with a transcription system including a speech processor linked to at least one speech recognition engine and at least one transcriptionist, the speech processor includes an input for receiving speech files and storage means for storing the received speech files until such a time that they are forwarded to a selected speech recognition engine or transcriptionist for processing, the performance gauge comprising:

a recognition engine text file database in which recognition engine text files are transcribed via the at least one speech recognition engine, each stored recognition engine text file including recognition engine operational information and a plurality of time stamps, whereby each time stamp indicates a position within a respective stored recognition engine text file;

a transcriptionist text file database in which manually transcribed speech files as manual transcriptionist text files and associated corrected recognition engine text files are stored, each manual transcriptionist text file including transcriptionist operational information and a plurality of time-stamps, each time-stamp indicative of a position within a respective speech file;

a comparator configured to receive a selected speech file, an associated recognition engine text file of said selected speech, and an associated manual transcriptionist text file of said selected speech file and determine a difference between corresponding time-stamps of said associated recognition text file and said associated manual transcriptionist text file with said associated manual transcriptionist text file as a baseline, wherein the comparator is further configured to determine a difference in time for preparing the associated manual transcriptionist text file and the associated corrected speech recognition engine text file; and

a report generator receiving the differences in the associated manual transcriptionist text file and the associated recognition engine text file and comparison of time between the associated recognition engine text file and the associated corrected recognition engine text file from said comparator so as to issue a report for evaluation by administrators of the transcription system.

2. The performance gauge according to claim 1, wherein the system includes a speech processor linked to a plurality of speech recognition engines and a plurality of transcriptionists.

3. The performance gauge according to claim 2, wherein recognition engine text files and transcriptionist files are linked within a database of the speech processor.

4. The performance gauge according to claim 1, wherein differences determined by the comparator further include additions, deletions, and mistranscriptions to the text files.

5. The performance gauge according to claim 1, wherein said recognition operational information comprises at least one of time-stamped data, a speech-recognition engine identification, and a processing time identification.

6. The performance gauge according to claim 1, wherein said transcriptionist operational information comprises at least one of time-stamped data, a transcriptionist processing time, a speech recognition engine identification data, transcriptionist identification data, and a processing time information.

7. A method for gauging performance within a transcription system including a speech processor linked to at least one speech recognition engine and at least one transcriptionist, the speech processor includes an input for receiving speech files and storage means for storing the received speech files until such a time that they are forwarded to a selected speech recognition engine or transcriptionist for processing, the method including the following steps of:

preparing a manual recognition text file based on a speech file, said manual recognition text file including transcriptionist operational information and a plurality of time-stamps, each time-stamp indicative of a position within the manual recognition text file;

preparing a recognition engine text file based upon automated speech recognition of the speech file via the at least one speech recognition engine, the recognition engine text file including recognition engine operational information and a plurality of time stamps, whereby each time-stamp indicative of a position within the recognition engine text file;

correcting the recognition engine text file to create a corrected recognition engine text file, the corrected recognition engine text file including transcriptionist operational information;

comparing the recognition engine text file and the manual recognition engine text file to determine differences between corresponding time stamps between the recognition engine text file and the manual recognition text file, wherein manual recognition text file is considered as a baseline and to determine a difference in time for preparing the manual recognition text file and the corrected recognition engine text file; and

compiling and reporting the differences for evaluation by administrators of the transcription system.

8. The method according to claim 7, wherein the step of correcting the recognition engine text file includes manually reviewing the recognition engine text file.

9. The method according to claim 8, wherein the step of manually reviewing the recognition engine text file includes listening to the speech file while reviewing the recognition engine text file and preparing the corrected recognition engine text file.

10. The method according to claim 7, wherein the transcription system includes the speech processor linked to a plurality of speech recognition engines and a plurality of transcriptionists.

11. The method according to claim 10, wherein recognition engine text files and manual text files are linked within a database of the speech processor.

12. The method according to claim 7, wherein the differences determined by the comparator further include additions, deletions, and mistranscriptions to the text files.

13. The method according to claim 7, wherein said recognition operational information comprises at least one of time-stamped data, a speech-recognition engine identification, and a processing time identification.

14. The method according to claim 7, wherein said transcriptionist operational information comprises at least one of time-stamped data, a transcriptionist processing time, a speech recognition engine identification data, transcriptionist identification data, and a processing time information.