System and method of automated evaluation of transcription quality

• US 9,747,890 B2
• Filed: 06/13/2016
• Issued: 08/29/2017
• Est. Priority Date: 07/30/2013
• Status: Active Grant

• Current Assignee: Verint Systems Incorporated
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Sidi, Oana, Wein, Ron
• Primary Examiner(s): SINGH, SATWANT K
• Application Number:
  US15/180,325
• Publication Number:
  US 20160365089A1
• Time in Patent Office: 442 Days
• Field of Search: None
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/12 using dynamic programming techniques, e.g. dynamic time warping [DTW]
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Systems and methods automatedly evaluate a transcription quality. Audio data is obtained. The audio data is segmented into a plurality of utterances with a voice activity detector operating on a computer processor. The plurality of utterances are transcribed into at least one word lattice with a large vocabulary continuous speech recognition system operating on the processor. A minimum Bayes risk decoder is applied to the at least one word lattice to create at least one confusion network. At least conformity ratio is calculated from the at least one confusion network.

Claims

1. A method of automated evaluation of a transcription quality, the method comprising:

obtaining audio data;

segmenting the audio data into a plurality of utterances with a voice activity detector operating on a computer processor;

transcribing the plurality of utterances into at least one word lattice with a large vocabulary continuous speech recognition system operating on the processor, wherein each of the plurality of utterances is transcribed by the processor into a respective word lattice;

applying, by the processor, a minimum Bayes risk decoder to the at least one word lattice to create at least one confusion network representing the at least one word lattice as a plurality sequential word bins and ε

-bins, wherein the processor creates a confusion network for each word lattice; and

calculating, with the processor, at least one conformity ratio from the least one confusion network, wherein the processor calculates a conformity ratio for each confusion network;

calculating, with the processor, a transcription quality score from the at least one conformity ratio;

filtering, by the processor, the plurality confusion networks based upon the calculated transcription plurality score of each confusion network;

selecting, by the processor, those confusion networks from the plurality of confusion networks having a transcription quality score greater than a predetermined value; and

storing, by the processor, the selected confusion networks as a plurality of high quality transcriptions.

2. The method of claim 1, wherein the audio data is streaming audio data.

3. The method of claim 1, wherein calculating the conformity ration for each confusion network further comprises:

identifying a probability value of a most probable word arc in each word bin; and

calculating a joint probability for each ε

-bin and a preceding word bin;

wherein the conformity ratio is an average of the calculated joint probabilities for the confusion network.

4. The method of claim 1, wherein the transcription quality score is a normalized value of the conformity ratio.

5. The method of claim 1, further comprising calculating an overall conformity ratio for a transcription of the audio data from the conformity ratios calculated from the confusion network of each of the utterances in the plurality of utterances.

6. The method of claim 1, further comprising creating a transcription model by analyzing the plurality of high quality transcriptions.

7. The method of claim 1, further comprising:

obtaining the utterances associated with each of the confusion networks in the plurality of high quality transcriptions; and

creating a transcription model based upon the obtained utterances.

8. The method of claim 1, further comprising producing an indication of the of the transcription quality score.

9. The method of claim 1, wherein transcribing the plurality of utterances comprises applying at least one transcription model to each of the plurality of utterances and wherein the at least one conformity ratio is indicative of a conformity between the audio data and the at least one transcription model.

10. The method of claim 9, further comprising:

selecting a new at least one transcription model based upon the at least one conformity ratio; and

transcribing the plurality of utterances by applying the new at least one transcription model to each of the plurality of utterances.

11. A system for automated evaluation of transcription quality, the system comprising:

an audio data source upon which a plurality of audio data files are stored;

a processor that receives the plurality of audio data files, segments the audio data files into a plurality of utterances and applies at least one transcription model to the plurality of utterances to transcribe the plurality of utterances into at least one word lattice, wherein each of the plurality of utterances are transcribed into a respective word lattice;

a non-transient computer readable medium communicatively connected to the processor and programmed with computer readable code that when executed by the processor causes the processor to;

apply a minimum Bayes risk decoder to the at least one word lattice to create at least one confusion network representing the at least one word lattice as a plurality of sequential word bins and ε

-bins, wherein a confusion network is created for each word lattice;

calculate at least one conformity ratio from the at least one confusion network, wherein a conformity ratio is calculated for each confusion network;

calculate a transcription quality score from the at least one conformity ratio

filter the plurality confusion networks based upon the calculated transcription quality score of each confusion network;

select those confusion networks from the plurality of confusion networks having a transcription quality score greater than a predetermined value; and

store the selected confusion networks as a plurality of high quality transcriptions.

12. A non-transient computer readable medium programmed with computer readable code that upon execution by as processor causes the processor to:

obtain audio data;

segment the audio data into a plurality of utterances with a voice activity detector;

transcribe the plurality of utterances into at least one word lattice with a large vocabulary continuous speech recognition system, each of the plurality of utterances being transcribed into a respective word lattice;

apply a minimum Bayes risk decoder to the at least one word lattice to create at least one confusion network representing the at least one word lattice as a plurality of sequential word bins and ε

-bins, wherein a confusion network is created for each word lattice;

calculate at least one conformity ratio from the at least one confusion network, wherein a conformity ratio is calculated for each confusion network;

calculate a transcription quality score from the at least one conformity ratio;

filter the plurality confusion networks based upon the calculated transcription quality score of each confusion network;

select those confusion networks from the plurality of confusion networks having a transcription quality score greater than a predetermined value; and

store the selected confusion networks as a plurality of high quality transcriptions.

13. The non-transient computer readable medium of claim 12, wherein execution of the computer readably code by the processor further causes the processor to:

identify a probability value of a most probable word arc in each word bin; and

calculate a joint probability for each ε

-bin and a preceding word bin;

wherein the conformity ratio is an average of the calculated joint probabilities for the confusion network.

14. The non-transient computer readable medium of claim 13, wherein at least one transcription model is applied to the plurality of utterances to transcribe the plurality of utterances and wherein the at least one conformity ratio is indicative of a conformity between the audio data and the at least one transcription model.