Method and system for predicting understanding errors in a task classification system

• US 6,751,591 B1
• Filed: 01/22/2001
• Issued: 06/15/2004
• Est. Priority Date: 01/22/2001
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Gorin, Allen Louis, Geary, Irene Langkilde, Walker, Marilyn Ann, Wright, Jeremy H.
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US09/765,393
• Time in Patent Office: 1,240 Days
• Field of Search: 704/250, 704/251, 704/254, 704/256, 704/257, 379/88.01
• US Class Current: 704/257
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

This invention concerns a method and system for monitoring an automated dialog system for the automatic recognition of language understanding errors based on a user'"'"'s input communications in a task classification system. The method may include determining whether the user'"'"'s input communication can be understood in order to make a task classification decision. If the user'"'"'s input communication cannot be understood and a task classification decision cannot be made, a probability of understanding the user'"'"'s input communication may be determined. If the probability exceeds a first threshold, further dialog may be conducted with the user. Otherwise, the user may be directed to a human for assistance. In another possible embodiment, the method operates as above except that if the probability exceeds a second threshold, the second threshold being higher than the first, then further dialog may be conducted with the user using the current dialog strategy. However, if the probability falls between a first threshold and a second threshold, the dialog strategy may be adapted in order to improve the chances of conducting a successful dialog with the user. This process may be cumulative. In particular, the first dialog exchange may be stored in a database. Then, a second dialog exchange is conducted with the user. As a result, a second determination is made as to whether the user'"'"'s input communication can be understood can be conducted based on the stored first exchange and the current second exchanges. This cumulative process may continue using a third and fourth exchange, if necessary.

Claims

1. A method of monitoring language understanding of a user'"'"'s input communication in a task classification system that operates on a task objective of a user, comprising:

determining whether the user'"'"'s input communication can be understood in order to make a task classification decision, wherein if the user'"'"'s input communication cannot be understood and a task classification decision cannot be made, the method comprises;

determining whether a probability of understanding the user'"'"'s input communication exceeds a first threshold, wherein if the first threshold is exceed, further dialog is conducted with the user, otherwise,

determining whether a probability of understanding the user'"'"'s input communication threshold exceeds a second threshold, the second threshold being greater than the first threshold, wherein if the second threshold is exceeded, further dialog is conducted with the user using a current dialog strategy, otherwise, further dialog is conducted with the user using an adapted dialog strategy.

2. The method of claim 1, wherein if the first threshold is not exceeded, the user is routed to a human for assistance.

3. The method of claim 1, wherein if the task classification decision can be made, the task is classified as one of a set of task objectives and the user'"'"'s input communication is routed in accordance with the classified task objective.

4. The method of claim 1, further comprising:

receiving the user'"'"'s input communication;

recognizing portions of the user'"'"'s input communication; and

providing an input to a language understanding monitor based on applying a confidence function to the recognized portions of the user'"'"'s input communications.

5. The method of claim 4, wherein the adapted dialog strategy includes one of prompting the user with choices and prompting the user to confirm the recognition and understanding data.

6. The method of claim 1, wherein the user'"'"'s input communication includes at least one of verbal and nonverbal communications.

7. The method of claim 6, wherein the nonverbal communications include at least one of gestures, body movements, head movements, non-responses, text, keyboard entries, mouse clocks, DTMF codes, pointers, stylus, cable set-top box entries graphical user interface entries, and touchscreen entries.

8. The method of claim 1, wherein the method is used for customer care purposes.

9. The method of claim 1, wherein the probability is determined using recognition and understanding data derived from the user'"'"'s input communication.

10. The method of claim 1, wherein the probability is determined using training data stored in a training database, the training data including at least one of classification models and extracted features.

11. The method of claim 10, wherein the extracted features are derived from recognition, understanding and dialog data.

12. A method of monitoring language understanding of a user'"'"'s input communication in a task classification system that operates on a task objective of a user, comprising:

determining whether the user'"'"'s input communication can be understood in order to make a task classification decision, wherein if the user'"'"'s input communication be understood and a task classification decision cannot be made, the method comprises;

determining whether a probability of understanding the user'"'"'s input communication exceeds a first threshold, wherein if the first threshold is exceeded, further dialog is conducted with the user

storing a first dialog exchange in a dialog history database, wherein the first dialog exchange includes a first automated dialog output and the user'"'"'s first input communication and the further dialog conducted with the user results in a second dialog exchange, wherein the second dialog exchange includes a second dialog output and the user'"'"'s second input communication; and

determining whether the probability of understanding exceeds the first threshold using the first dialog exchange and the second dialog exchange.

13. The method of claim 12, wherein the method is recursive in that the determining step determines whether the probability of understanding exceeds the first threshold using the each of the dialog exchanges conducted.

14. A language understanding monitoring system that operates in a task classification system, comprising:

a task classification processor that determines whether a task classification decision can be made based on an understanding of a user'"'"'s input communication;

a dialog manager that outputs dialog to the user;

a language understanding monitor that determines whether a probability of understanding the user'"'"'s input communication exceeds a first threshold, wherein if the first threshold is exceeded, the language understanding monitor prompts the dialog manager to conduct further dialog with the user;

a dialog history database that stores a first dialog exchange, wherein the first dialog exchange includes a first automated dialog output and the user'"'"'s first input communication, and the further dialog conducted with the user results in a second dialog exchange, the second dialog exchange linking a second dialog output and the user'"'"'s second input communication, and the language understanding monitor determining whether the probability of understanding exceeds the first threshold using the first dialog exchange and the second dialog exchange.

15. The system of claim 14, wherein the language understanding monitor determines whether the probability of understanding exceeds the first threshold using the each of the dialog exchanges conducted.

16. A language understanding monitoring system that operates in a task classification system, comprising:

a task classification processor that determines whether a task classification decision can be made based on an understanding of a user'"'"'s input communication;

a dialog manager that outputs dialog to the user;

a language understanding monitor that determines whether a probability of understanding the user'"'"'s input communication exceeds a first threshold, wherein if the first threshold is exceeded, the language understanding monitor prompts the dialog manager to conduct further dialog with the user, otherwise,

the language understanding monitor determines whether a probability of understanding the user'"'"'s input communication exceeds a second threshold, the second threshold being greater than the first threshold, and if the second threshold is exceed, the language understanding monitor prompts the dialog manager to conduct further dialog with the user using a current dialog strategy, wherein if the second threshold is not exceeded, the language understanding monitor prompts the dialog manager to conduct further dialog with the user using an adapted dialog strategy.

17. The system of claim 16, wherein if the first threshold is not exceeded, the language understanding monitor prompts the dialog manager to route the user to a human for assistance.

18. The system of claim 16, further comprising:

a recognizer that recognizes the user'"'"'s input communication; and

a language understanding unit that applies a confidence function to the recognized portions of the user'"'"'s input communication and provides an input to the language understanding monitor.

19. The system of claim 18, wherein the language understanding monitor determines the probability using recognition data from the recognizer and understanding data from the language understanding unit derived from the user'"'"'s input communication.

20. The system of claim 16, wherein the adapted dialog strategy includes one of prompting the user with choices and prompting the user to confirm the recognition and understanding data.

21. The system of claim 16, wherein the user'"'"'s input communication includes at least one of verbal and nonverbal communications.

22. The system of claim 16, wherein the system is used for customer care purposes.

23. The system of claim 16, wherein the language understanding monitor determines the probability using recognition data provided by a recognizer and understanding data provided by a language understanding unit, and the recognition and understanding data is derived from the user'"'"'s input communication.

24. The system of claim 16, further comprising a training database for storing training data for language understanding, wherein the language understanding monitor determines the probability using the training data stored in the training database, the training data including at least one of classification models and extracted features.

25. The system of claim 24, wherein the extracted features are derived from recognition, understanding and dialog data.

26. The system of claim 16, wherein if the task classification processor can make a task classification decision, the task classification processor classifies the task as one of a set of task objectives and routes the user'"'"'s input communication in accordance with the classified task objective.

27. A method of monitoring language understanding of a user'"'"'s input communication in a task classification system that operates on a task objective of a user, comprising:

determining whether the user'"'"'s input communication can be understood in order to make a task classification decision, wherein if the user'"'"'s input communication cannot be understood and a task classification decision cannot be made, the method comprises;

determining whether a probability of understanding the user'"'"'s input communication exceeds a first threshold, wherein if the first threshold is exceeded, further dialog is conducted with the user, otherwise, the user is routed to a human for assistance;

determining whether a probability of understanding the user'"'"'s input communication exceeds a second threshold, the second threshold being greater than the first threshold, wherein if the second threshold is exceeded, further dialog is conducted with the user using a current dialog strategy, otherwise further dialog is conducted with the user using an adapted dialog strategy.