Computer-implemented system and method for transcription error reduction

• US 10,083,691 B2
• Filed: 12/29/2017
• Issued: 09/25/2018
• Est. Priority Date: 07/20/2010
• Status: Active Grant

• Current Assignee: Avaya LLC (Avaya Incorporated), Avaya Management L.P. (Avaya Incorporated)
• Original Assignee: Intellisist, Inc. dba Spoken Communications (Avaya Incorporated)
• Inventors: Milstein, David
• Primary Examiner(s): YEN, ERIC L
• Application Number:
  US15/859,176
• Publication Number:
  US 20180122382A1
• Time in Patent Office: 270 Days
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/0631 Creating reference templates; Clustering
• CPC Class Codes: G10L 2015/0638 Interactive procedures
• CPC Class Codes: H04M 3/42221 Conversation recording systems at the subscriber's set H04M1/656
• CPC Class Codes: H04M 3/51 Centralised call answering arrangements requiring operator intervention , e.g. call or contact centers for telemarketing
• CPC Class Codes: H04M 3/5183 Call or contact centers with computer-telephony arrangements

Abstract

A computer-implemented system and method for transcription error reduction is provided. A transcribed value and a confidence score are assigned to each utterance in a stream of audio data. Those utterances with confidence scores that fall below a confidence threshold are identified as questionable utterances. At least one of the questionable utterances is placed into a pool of related utterances also determined to be questionable. A determination is made as to whether the pool satisfies a size threshold within a predetermined amount of time. A sample of the questionable utterances from the pool is obtained when the pool satisfies the size threshold within the predetermined amount of time. The sample of questionable utterances is provided to at least one human transcriber for verification.

Claims

1. A computer-implemented system for transcription error reduction, comprising:

a database to store a stream of audio data comprising utterances, wherein each utterance is assigned a transcribed value and a confidence score; and

a server comprising a central processing unit, memory, an input port to receive the utterances, and an output port, wherein the central processing unit is configured to perform the following steps;

identify those utterances with confidence scores that fall below a confidence threshold as questionable utterances;

place at least one of the questionable utterances into a pool of related utterances also determined to be questionable;

determine whether the pool satisfies a size threshold within a predetermined amount of time;

obtain a sample of the questionable utterances in the pool when the pool satisfies the size threshold within the predetermined amount of time; and

provide the sample of questionable utterances to at least one human transcriber for verification, wherein the steps reduce an amount of manual transcription required by a transcription system.

2. A system according to claim 1, wherein the central processing unit selects the sample of questionable utterances one of randomly and based on specific parameters.

3. A system according to claim 1, wherein the verification by the human transcriber comprises one of confirming one or more of the transcribed values and assigning a different transcribed value to one or more of the questionable utterances in the sample.

4. A system according to claim 3, wherein the central processing unit compares the confirmed transcribed values and the different transcribed values of the questionable utterances in the sample and when the confirmed transcribed values and the different transcribed values are the same, assigns the same transcribed value to the questionable utterances in the pool that are not included in the sample.

5. A system according to claim 3, wherein the central processing unit compares the confirmed transcribed values and the different transcribed values of the questionable utterances in the sample and when the confirmed assigned transcribed values and the different transcribed values differ, provides the questionable utterances in the pool that are not included in the sample to a different human reviewer.

6. A system according to claim 3, wherein the central processing unit incorporates each questionable utterance and the confirmed transcribed value or the different transcribed value for that questionable utterance into a grammar.

7. A system according to claim 1, wherein the central processing unit determines the related questionable utterances in the pool based on one or more of the transcribed values, the confidence scores, and a range of similarity shared between two or more of the related questionable utterances.

8. A system according to claim 1, wherein the central processing unit places another one of the questionable utterances into a pool of related utterances also determined to be questionable, determines whether the pool satisfies a size threshold within a predetermined amount of time, and provides all the questionable utterances in the pool to a human reviewer when the pool does not satisfy the size threshold within the predetermined amount of time.

9. A system according to claim 1, wherein the central processing unit adjusts one or more of a size of the sample, the size threshold of the pool, and the predetermined time based on a type of the stream of the audio data and an accuracy of transcription required of the audio data.

10. A system according to claim 1, wherein the audio data stream is obtained from one of a live call and a recording.

11. A computer-implemented method for transcription error reduction, comprising the steps of:

assigning a transcribed value and a confidence score to each utterance in a stream of audio data;

identifying those utterances with confidence scores that fall below a confidence threshold as questionable utterances;

placing at least one of the questionable utterances into a pool of related utterances also determined to be questionable;

determining whether the pool satisfies a size threshold within a predetermined amount of time;

obtaining a sample of the questionable utterances in the pool when the pool satisfies the size threshold within the predetermined amount of time; and

providing the sample of questionable utterances to at least one human transcriber for verification, wherein the steps reduce an amount of manual transcription required by a transcription system.

12. A method according to claim 11, further comprising:

selecting the sample of questionable utterances one of randomly and based on specific parameters.

13. A method according to claim 11, wherein the verification by the human transcriber comprises one of confirming one or more of the transcribed values and assigning a different transcribed value to one or more of the questionable utterances in the sample.

14. A method according to claim 13, further comprising:

comparing the confirmed transcribed values and the different transcribed values of the questionable utterances in the sample; and

when the confirmed transcribed values and the different transcribed values are the same, assigning the same transcribed value to the questionable utterances in the pool that are not included in the sample.

15. A method according to claim 13, further comprising:

comparing the confirmed transcribed values and the different transcribed values of the questionable utterances in the sample; and

when the confirmed transcribed values and the different transcribed values differ, providing the questionable utterances in the pool that are not included in the sample to a different human reviewer.

16. A method according to claim 13, further comprising:

incorporating each questionable utterance and the confirmed transcribed value or the different transcribed value for that questionable utterance into a grammar.

17. A method according to claim 11, further comprising:

determining the related questionable utterances in the pool based on one or more of the transcribed values, the confidence scores, and a range of similarity shared between two or more of the related questionable utterances.

18. A method according to claim 11, further comprising:

placing another one of the questionable utterances into a pool of related utterances also determined to be questionable;

determining whether the pool satisfies a size threshold within a predetermined amount of time;

providing all the questionable utterances in the pool to a human reviewer when the pool does not satisfy the size threshold within the predetermined amount of time.

19. A method according to claim 11, further comprising:

adjusting one or more of a size of the sample, the size threshold of the pool, and the predetermined time based on a type of the stream of the audio data and an accuracy of transcription required of the audio data.

20. A method according to claim 11, wherein the audio data stream is obtained from one of a live call and a recording.