Method and system for ensuring robustness in natural language understanding

• US 7,016,827 B1
• Filed: 09/03/1999
• Issued: 03/21/2006
• Est. Priority Date: 09/03/1999
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Ramaswamy, Ganesh N., Kleindienst, Jan, Jamieson, Kyle A.
• Primary Examiner(s): Knepper, David D.
• Assistant Examiner(s): Han, Qi
• Application Number:
  US09/390,308
• Time in Patent Office: 2,391 Days
• Field of Search: 704/1, 704/9, 704/270, 704/275, 704/257
• US Class Current: 704/9
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A method and system, which may be implemented by employing a program storage device readable by machine, and tangibly embodying a program of instructions executable by the machine to perform method steps for ensuring robustness of a natural language understanding (NLU) system, includes tagging recognized words of a command input to the NLU system to associate the command with a context, and translating the command to at least one formal command based on the tagged words. A top ranked formal command is determined based on scoring of the tagged recognized words and scoring translations of the at least one formal command. Whether the top ranked formal command is accepted is determined by comparing a feature vector of the top ranked formal command to representations of feature vectors stored in an accept model. The top ranked formal command is executed if accepted and incorrect commands are prevented from execution to provide a robust NLU system.

Claims

1. A method for ensuring robustness of a natural language understanding (NLU) system comprising the steps of:

tagging recognized words of a command input to the NLU system to associate the command with a context;

translating the command to at least one formal command based on the tagged words;

determining a top ranked formal command based on scoring of the tagged recognized words and scoring translations of the at least one formal command;

determining whether the top ranked formal command is accepted by comparing a feature vector of the top ranked formal command to representations of feature vectors stored in an accept model; and

executing the top ranked formal command if accepted and preventing incorrect commands from execution to provide a robust NLU system.

2. The method as recited in claim 1, wherein the step of determining a top ranked formal command includes the step of ranking formal commands based on a product of scores of the tagged words and scores of translations of the at least one formal command.

3. The method as recited in claim 1, wherein the step of determining a top ranked formal command includes the step of ranking N formal commands where N is a selectable system parameter.

4. The method as recited in claim 1, further comprising the step of determining whether the top ranked formal command is rejected by comparing the feature vector of the top ranked formal command to representations of feature vectors stored in a reject model.

5. The method as recited in claim 4, further comprising the step of providing the reject model by including representations of feature vectors of formal commands corresponding to words or sentences to be rejected.

6. The method as recited in claim 4, wherein the reject model includes a cluster of models.

7. The method as recited in claim 6, further comprising the step of clustering the cluster of models based on at least one of mistakes in commands, mistakes in arguments of the command, and processing mistakes.

8. The method as recited in claim 1, further comprising the step of providing the accept model by including representations of feature vectors of formal commands corresponding to words or sentences to be accepted.

9. The method as recited in claim 1, wherein the step of determining whether the top ranked formal command is accepted includes the step of computing a probability of acceptance for the command.

10. The method as recited in claim 9, wherein the step of computing a probability of acceptance for the command includes the steps of:

computing a probability of rejection for the command and comparing the probability of acceptance to the probability of rejection to determine if the command is to be executed.

11. The method as recited in claim 9, wherein the step of computing a probability of acceptance for the command includes the step of comparing the probability of acceptance to a threshold probability to determine if the command is to be executed.

12. The method as recited in claim 11, further comprising the step of modifying the threshold by the user.

13. The method as recited in claim 1, wherein the accept model includes a cluster of models.

14. The method as recited in claim 13, further comprising the step of clustering the cluster of models based on at least one of mistakes in commands, mistakes in arguments of the command, and processing mistakes.

15. The method as recited in claim 1, wherein the step of preventing incorrect commands, includes executing a do nothing command responsive to the incorrect commands.

16. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for ensuring robustness of a natural language understanding (NLU) system, the method steps comprising:

tagging recognized words of a command input to the NLU system to associate the command with a context;

translating the command to at least one formal command based on the tagged words;

determining a top ranked formal command based on scoring of the tagged recognized words and scoring translations of the at least one formal command;

determining whether the top ranked formal command is accepted by comparing a feature vector of the top ranked formal command to representations of feature vectors stored in an accept model; and

executing the top ranked formal command if accepted and preventing incorrect commands from execution to provide a robust NLU system.

17. The program storage device as recited in claim 16, wherein the step of determining a top ranked formal command includes the step of ranking formal commands based on a product of scores of the tagged words and scores of translations of the at least one formal command.

18. The program storage device as recited in claim 16, wherein the step of determining a top ranked formal command includes the step of ranking N formal commands where N is a selectable system parameter.

19. The program storage device as recited in claim 16, further comprising the step of determining whether the top ranked formal command is rejected by comparing the feature vector of the top ranked formal command to representations of feature vectors stored in a reject model.

20. The program storage device as recited in claim 19, further comprising the step of providing the reject model by including representations of feature vectors of formal commands corresponding to words or sentences to be rejected.

21. The program storage device as recited in claim 19, wherein the reject model includes a cluster of models.

22. The program storage device as recited in claim 21, further comprising the step of clustering the cluster of models based on at least one of mistakes in commands, mistakes in arguments of the command, and processing mistakes.

23. The program storage device as recited in claim 16, further comprising the step of providing the accept model by including representations of feature vectors of formal commands corresponding to words or sentences to be accepted.

24. The program storage device as recited in claim 16, wherein the step of determining whether the top ranked formal command is accepted includes the step of computing a probability of acceptance for the command.

25. The program storage device as recited in claim 24, wherein the step of computing a probability of acceptance for the command includes the steps of:

computing a probability of rejection for the command and comparing the probability of acceptance to the probability of rejection to determine if the command is to be executed.

26. The program storage device as recited in claim 24, wherein the step of computing a probability of acceptance for the command includes the step of comparing the probability of acceptance to a threshold probability to determine if the command is to be executed.

27. The program storage device as recited in claim 26, further comprising the step of modifying the threshold by the user.

28. The program storage device as recited in claim 16, wherein the accept model includes a cluster of models.

29. The program storage device as recited in claim 28, further comprising the step of clustering the cluster of models based on at least one of mistakes in commands, mistakes in arguments of the command, and processing mistakes.

30. The program storage device as recited in claim 16, wherein the step of preventing incorrect commands, includes executing a do nothing command responsive to the incorrect commands.

31. A natural language understanding (NLU) system comprising:

means for tagging recognized words of a command input to the NLU system to associate the command with a context;

means for translating the command to at least one formal command based on the tagged words;

a robustness checker for determining a top ranked formal command based on scoring of the tagged recognized words and scoring translations the at least one formal command, the robustness checker for determining whether the top ranked formal command is accepted by comparing a feature vector of the top ranked formal command to representations feature vectors stored in an accept model; and

a command executor for executing the top ranked formal command if accepted and preventing incorrect commands from execution to provide a robust NLU system.

32. The system as recited in claim 31, wherein the top ranked formal command is ranked based on a product of scores of the tagged words and scores of translations of the at least one formal command.

33. The system as recited in claim 31, wherein the top ranked formal command is selected from a ranked list of N formal commands where N is a selectable system parameter.

34. The system as recited in claim 31, further comprising an accept model for storing representations feature vectors of accept commands, the representations of feature vectors being employed to compare with a user input command to determine an acceptance probability of formal commands.

35. The method as recited in claim 34, wherein the accept model includes a cluster of models.

36. The system as recited in claim 31, further comprising a reject model for storing representations of feature vectors of reject commands, the feature vectors being employed to compare with a user input command to determine a rejection probability of formal commands.

37. The system as recited in claim 36, wherein the reject model includes a cluster of models.

38. The system as recited in claim 31, wherein the robustness checker includes a feature extractor for extracting feature vectors from the command.

39. The system as recited in claim 31, wherein the feature vectors include tagging scores for recognized words of a word or sentence represented by the feature vector and translation scores for formal commands associated with the word or sentence represented by the feature vector.

40. The system as recited in claim 31, wherein the feature vectors include a do nothing score associated with words and sentences, the do nothing score for indicating a probability that a do nothing command is present for associated words and sentences, the do nothing command being associated with incomplete or incorrect commands which may be input as a command.

41. The system as recited in claim 31, wherein the feature vectors include at least one of a top command similarity measure for counting identical formal commands and a parameter mismatch feature for measuring a number of command arguments in a translation of a command.

42. The system as recited in claim 31, wherein the robustness checker includes a robustness evaluator for determining whether the top ranked formal command is accepted by comparing the top ranked formal command to feature vectors stored in an accept model.