System and Method of Determining a Domain and/or an Action Related to a Natural Language Input

• US 20160148612A1
• Filed: 11/24/2015
• Published: 05/26/2016
• Est. Priority Date: 11/26/2014
• Status: Active Grant

• Current Assignee: Cerence Inc.
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: GUO, Yikun, SALEEM, Safiyyah, HUANG, Jiayuan
• Granted Patent: US 10,431,214 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

The disclosure relates to methods, systems and other embodiments directed to determining an information domain match for a natural language (NL) input (e.g., a spoken utterance), and confirming whether the NL input is correctly matched to the information domain. For example, after receiving an NL input, a first information domain to which the NL input belongs and a feature value set may be determined based on a semantic pattern matching technique. Further, a second information domain to which the NL input belongs, and a corresponding confidence score related to the second information domain may be determined. The second information domain may be determined based on a first statistical classification technique. Based on the determined feature value set and the confidence score related to the second information domain, it may be confirmed whether the NL input correctly belongs to the first information domain, e.g., based on a second statistical classification technique.

Claims

1. A method to confirm whether a domain match determined for a natural language input is correct, the method being implemented on a computer system having one or more physical processors programmed with computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the computer system, a natural language (NL) input;

determining, by the computer system, based on a semantic pattern matching technique, a first information domain to which the NL input belongs, and a feature value set;

determining, by the computer system, a second information domain to which the NL input belongs, and a corresponding confidence score related to the second information domain; and

determining, by the computer system, whether the NL input correctly belongs to the first information domain based at least on the feature value set and the confidence score.

2. The method of claim 1, wherein the NL input is represented based on one or more tokens, and

wherein said determining based on the semantic pattern matching technique comprises matching, by the computer system, based at least on a sematic grammar, the one or more tokens to one or more computer-executable tasks and one or more entities, wherein the one or more tasks and the one or more entities are predefined for the semantic pattern matching technique.

3. The method of claim 2, wherein the feature value set comprises one or more of the following:

a total number of the one or more tokens matched, a fraction number of the one or more tokens matched, a matching score assigned to at least one of the one or more tokens, and a matching score assigned to the NL input.

4. The method of claim 1, wherein the NL input is represented based on one or more tokens, the method further comprising generating, by the computer system, an input feature vector for the NL input, wherein each element of the input feature vector represents at least one of the one or more tokens,

wherein said determining the second information domain and the confidence score comprises processing, by the computer system, the input feature vector based on a first statistical classification technique.

5. The method of claim 4, wherein said processing the input feature vector comprises processing, by the computer system, the input feature vector based on the first statistical classification technique to determine a plurality of second information domains to which the NL input belongs and a corresponding plurality of confidence scores, wherein each confidence score is indicative of the likelihood of the NL input belonging to the corresponding second information domain.

6. The method of claim 4, further comprising processing, by the computer system, the input feature vector based on the first statistical classification technique to determine a binary check value indicative of whether the second information domain is related to one of a plurality of information domains associated with the semantic pattern matching technique,

wherein said determining whether the NL input correctly belongs to the first information domain is further based on the binary check value.

7. The method of claim 1, wherein said determining whether the NL input correctly belongs to the first information domain comprises processing, by the computer system, the feature value set and the confidence score based on a second statistical classification technique.

8. The method of claim 7, wherein said processing the feature value set and the confidence score based on the second statistical classification technique comprises:

obtaining, by the computer system, coefficient weights associated with the second statistical classification technique for the feature value set and the confidence score;

processing, by the computer system, the feature value set and the confidence score with the coefficient weights based on the second statistical classification technique to determine a confirmation score related to the first information domain; and

determining, by the computer system, whether the NL input correctly belongs to the first information domain based on a result of a comparison of the confirmation score related to the first information domain with a certain threshold.

9. The method of claim 4, wherein the first statistical classification technique is based on one or more of the following:

a maximum-entropy classifier, a logistic regression classifier, a Naï

ve Bayes classifier, and a support vector machine.

10. The method of claim 1, wherein the NL input comprises a spoken NL utterance.

11. A system to confirm whether a domain match determined for a natural language input is correct, the system comprising:

one or more physical processors programmed with computer program instructions which, when executed, cause the one or more physical processors to;

receive a natural language (NL) input;

determine, based on a semantic pattern matching technique, a first information domain to which the NL input belongs, and a feature value set;

determine a second information domain to which the NL input belongs, and a corresponding confidence score related to the second information domain; and

determine whether the NL input correctly belongs to the first information domain based at least on the feature value set and the confidence score.

12. The system of claim 11, wherein the NL input is represented using one or more tokens, and

wherein, to determine based on the semantic pattern matching technique, the one or more physical processors are further caused to match, based at least on a sematic grammar, the one or more tokens to one or more computer-executable tasks and one or more entities, wherein the one or more tasks and the one or more entities are predefined for the semantic pattern matching technique.

13. The system of claim 12, wherein the feature value set comprises one or more of the following:

a total number of the one or more tokens matched, a fraction number of the one or more tokens matched, a matching score assigned to at least one of the one or more tokens, and a matching score assigned to the NL input.

14. The system of claim 11, wherein the NL input is represented based on one or more tokens, and wherein the one or more physical processors are further caused to:

generate an input feature vector for the NL input, wherein each element of the input feature vector represents at least one of the one or more tokens, and

to determine the second information domain and the confidence score, process the input feature vector based on a first statistical classification technique.

15. The system of claim 14, wherein, to process the input feature vector, the one or more physical processors are further caused to process the input feature vector based on the first statistical classification technique to determine a plurality of second information domains to which the NL input belongs and a corresponding plurality of confidence scores, wherein each confidence score is indicative of the likelihood of the NL input belonging to the corresponding second information domain.

16. The system of claim 11, wherein the one or more physical processors are further caused to process the input feature vector based on the first statistical classification technique to determine a binary check value indicative of whether the second information domain is related to one of a plurality of information domains associated with the semantic pattern matching technique,

wherein, a determination of whether the NL input correctly belongs to the first information domain is further based on the binary check value.

17. The system of claim 11, wherein, to determine whether the NL input correctly belongs to the first information domain, the one or more physical processors are further caused to process the feature value set and the confidence score based on a second statistical classification technique.

18. The system of claim 17, wherein, to process the feature value set and the confidence score based on the second statistical classification technique, the one or more physical processors are further caused to:

obtain coefficient weights associated with the second statistical classification technique for the feature value set and the confidence score;

process the feature value set and the confidence score with the coefficient weights based on the second statistical classification technique to determine a confirmation score related to the first information domain; and

determine whether the NL input correctly belongs to the first information domain based on a result of a comparison of the confirmation score related to the first information domain with a certain threshold.

19. The system of claim 14, wherein the first statistical classification technique is based on one or more of the following:

a maximum-entropy classifier, a logistic regression classifier, a Naï

ve Bayes classifier, and a support vector machine.

20. The system of claim 11, wherein the NL input comprises a spoken NL utterance.

21. A method to determine a domain match for a natural language input, the method being implemented on a computer system having one or more physical processors programmed with computer program instructions which, when executed, perform the method, the method comprising:

receiving, at the computer system, a natural language (NL) input;

determining, by the computer system, a plurality of information domains and a plurality of confidence scores related to the NL input;

performing, by the computer system, a database search using the NL input as a search query, wherein the database search generates search results;

determining, by the computer system, a final information domain from the plurality of information domains for the NL input based on the plurality of confidence scores and information related to the search results.

22. The method of claim 21, further comprising assigning, by the computer system, ranking scores to the search results, wherein the ranking scores indicate a degree of relevance of the search results to the NL input,

wherein said determining, by the computer system, a final information domain is based on the plurality of confidence scores and the ranking scores.