SYSTEM AND METHOD FOR LOCALIZED ERROR DETECTION OF RECOGNITION RESULTS

• US 20160155445A1
• Filed: 12/01/2014
• Published: 06/02/2016
• Est. Priority Date: 12/01/2014
• Status: Active Grant

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Selfridge, Ethan, Johnston, Michael J., STOYANCHEV, Svetlana
• Granted Patent: US 9,953,644 B2
• Time in Patent Office: Days
• CPC Class Codes: G10L 15/01 Assessment or evaluation of speech recognition systems
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A system, method and computer-readable storage devices are disclosed for using targeted clarification (TC) questions in dialog systems in a multimodal virtual agent system (MVA) providing access to information about movies, restaurants, and musical events. In contrast with open-domain spoken systems, the MVA application covers a domain with a fixed set of concepts and uses a natural language understanding (NLU) component to mark concepts in automatically recognized speech. Instead of identifying an error segment, localized error detection (LED) identifies which of the concepts are likely to be present and correct using domain knowledge, automatic speech recognition (ASR), and NLU tags and scores. If at least concept is identified to be present but not correct, the TC component uses this information to generate a targeted clarification question. This approach computes probability distributions of concept presence and correctness for each user utterance, which can apply to automatic learning for clarification policies.

Claims

1. A method comprising:

processing, via a speech recognizer, an utterance from a speaker to produce speech recognition output;

identifying speech segments in the speech recognition output;

generating a concept presence score and a concept correctness score for each speech segment using context that is unavailable to the speech recognizer;

selecting one of the speech segments below a certainty threshold;

generating, for the one of the speech segments and based on the concept presence score and the concept correctness score, a targeted clarification question associated with the utterance; and

presenting the targeted clarification question to the speaker in response to the utterance.

2. The method of claim 1, wherein the context that is unavailable to the speech recognizer comprises one of dialog history, concept co-occurrence probability, domain history, speech recognition confidence scores, contextual features of the utterance, and tagging scores.

3. The method of claim 1, wherein the concept presence score indicates a confidence that a concept type is present in a respective speech segment, and wherein the concept correctness score indicates a confidence that an identification of the concept type is correct.

4. The method of claim 1, wherein the targeted clarification question is generated based on a question template associated with the speech segments.

5. The method of claim 1, further comprising:

identifying multiple speech segments below the certainty threshold; and

generating the targeted clarification question based on respective concept presence scores and concept correctness scores for the multiple speech segments.

6. The method of claim 1, wherein the concept presence score and the concept correctness score are generated based on a domain of available concepts.

7. The method of claim 1, wherein the speech recognizer identifies that at least one of the segments has a concept presence score above the certainty threshold.

8. A system comprising:

a processor;

a speech recognizer; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

processing, via a speech recognizer, an utterance from a speaker to produce speech recognition output;

identifying speech segments in the speech recognition output;

generating a concept presence score and a concept correctness score for each speech segment using context that is unavailable to the speech recognizer;

selecting one of the speech segments below a certainty threshold;

generating, for the one of the speech segments and based on the concept presence score and the concept correctness score, a targeted clarification question associated with the utterance; and

presenting the targeted clarification question to the speaker in response to the utterance.

9. The system of claim 8, wherein the context that is unavailable to the speech recognizer comprises one of dialog history, concept co-occurrence probability, domain history, speech recognition confidence scores, contextual features of the utterance, and tagging scores.

10. The system of claim 8, wherein the concept presence score indicates a confidence that a concept type is present in a respective speech segment, and wherein the concept correctness score indicates a confidence that an identification of the concept type is correct.

11. The system of claim 8, wherein the targeted clarification question is generated based on a question template associated with the speech segments.

12. The system of claim 8, the computer-readable storage medium having additional instructions stored which, when executed by the processor, cause the processor to perform operations comprising:

identifying multiple speech segments below the certainty threshold; and

generating the targeted clarification question based on respective concept presence scores and concept correctness scores for the multiple speech segments.

13. The system of claim 8, wherein the concept presence score and the concept correctness score are generated based on a domain of available concepts.

14. The system of claim 8, wherein the speech recognizer identifies that at least one of the segments has a concept presence score above the certainty threshold.

15. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

processing, via a speech recognizer, an utterance from a speaker to produce speech recognition output;

identifying speech segments in the speech recognition output;

generating a concept presence score and a concept correctness score for each speech segment using context that is unavailable to the speech recognizer;

selecting one of the speech segments below a certainty threshold;

generating, for the one of the speech segments and based on the concept presence score and the concept correctness score, a targeted clarification question associated with the utterance; and

presenting the targeted clarification question to the speaker in response to the utterance.

16. The computer-readable storage device of claim 15, wherein the context that is unavailable to the speech recognizer comprises one of dialog history, concept co-occurrence probability, domain history, speech recognition confidence scores, contextual features of the utterance, and tagging scores.

17. The computer-readable storage device of claim 15, wherein the concept presence score indicates a confidence that a concept type is present in a respective speech segment, and wherein the concept correctness score indicates a confidence that an identification of the concept type is correct.

18. The computer-readable storage device of claim 15, wherein the targeted clarification question is generated based on a question template associated with the speech segments.

19. The computer-readable storage device of claim 15, having additional instructions stored which, when executed by the computing device, cause the computing device to perform operations comprising:

identifying multiple speech segments below the certainty threshold; and

generating the targeted clarification question based on respective concept presence scores and concept correctness scores for the multiple speech segments.

20. The computer-readable storage device of claim 15, wherein the concept presence score and the concept correctness score are generated based on a domain of available concepts.