Hybrid learning system for natural language intent extraction from a dialog utterance

• US 10,713,441 B2
• Filed: 01/02/2019
• Issued: 07/14/2020
• Est. Priority Date: 03/23/2018
• Status: Active Grant

• Current Assignee: ServiceNow Incorporated
• Original Assignee: ServiceNow Incorporated
• Inventors: Sapugay, Edwin, Madamala, Anil Kumar, Naboka, Maxim, Sunkara, Srinivas SatyaSai, Santos, Lewis Savio Landry, Subbarao, Murali B.
• Primary Examiner(s): Wozniak, James S
• Application Number:
  US16/238,324
• Publication Number:
  US 20190295535A1
• Time in Patent Office: 559 Days
• Field of Search: 704 9, 704232, 704257, 704275
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

An agent automation system includes a memory configured to store a natural language understanding (NLU) framework and a processor configured to execute instructions of the NLU framework to cause the agent automation system to perform actions. These actions comprise: generating an annotated utterance tree of an utterance using a combination of rules-based and machine-learning (ML)-based components, wherein a structure of the annotated utterance tree represents a syntactic structure of the utterance, and wherein nodes of the annotated utterance tree include word vectors that represent semantic meanings of words of the utterance; and using the annotated utterance tree as a basis for intent/entity extraction of the utterance.

Claims

1. An agent automation system, comprising:

a memory configured to store a natural language understanding (NLU) framework and an intent/entity model that includes written sample utterances; and

a processor configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising;

generating annotated utterance trees for a written user utterance and for the written sample utterances of the intent/entity model using a combination of rules-based and machine-learning (ML)-based components, wherein each annotated utterance tree includes nodes arranged in a dependency parse tree structure that represents a syntactic structure of a corresponding utterance, and wherein each of the nodes includes a word vector representing a semantic meaning of a word or phrase of the corresponding utterance;

generating a subtree vector for each subtree of the annotated utterance trees based on the word vectors of the nodes of each subtree of the annotated utterance trees; and

extracting an intent and/or entity from the written user utterance based on a comparison of the subtree vectors of the annotated utterance trees of the written user utterance to the subtree vectors of the annotated utterance trees of the written sample utterances.

2. The agent automation system of claim 1, wherein the NLU framework includes a vocabulary subsystem, a structure subsystem, and a prosody subsystem that cooperate to generate the annotated utterance tree of the corresponding utterance using the combination of rules-based and machine-learning (ML)-based components.

3. The agent automation system of claim 2, wherein the vocabulary subsystem includes a word vector distribution model, wherein word vector distribution model provides the word vectors for the nodes of the annotated utterance tree based on the word vector distribution model.

4. The agent automation system of claim 3, wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to improve operation of the vocabulary subsystem by performing actions comprising:

performing rule-based unsupervised learning of the words and/or phrases of the written user utterance; and

modifying word vectors of the word vector distribution model based on the rule-based unsupervised learning.

5. The agent automation system of claim 2, wherein the structure subsystem includes one or more rules-based parsers and one or more ML-based parsers, wherein the structure subsystem is configured to generate and annotate the structure of the annotated utterance trees based on the syntactic structures of their corresponding utterances.

6. The agent automation system of claim 5, wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system to improve operation of the one or more ML-based parsers by performing actions comprising:

parsing the written user utterance using the one or more rules-based parsers of the structure subsystem to generate a plurality of annotated utterance trees; and

in response to the processor determining that a majority of the plurality of annotated utterance trees have a common structure, adjusting at least one weight of one or more neural network models associated with the one or more ML-based parsers such that the one or more ML-based parsers are configured to generate the common structure from the written user utterance.

7. The agent automation system of claim 2, wherein the prosody subsystem includes one or more rules-based prosody systems and one or more ML-based prosody systems, wherein the prosody system is configured to segment the written user utterance based on written prosody cues of the written user utterance, wherein the written prosody cues comprise a rhythm, an emphasis, or a focus of the written user utterance.

8. The agent automation system of claim 1, wherein, to extract the intent and/or entity from the written user utterance, the processor is configured to execute instructions of the NLU framework to cause the agent automation system to perform actions comprising:

comparing the subtree vectors of the annotated utterance trees of the written user utterance to the subtree vectors of the annotated utterance trees of the written sample utterances of the intent/entity model to identify a matching written sample utterance;

determining the intent and/or entity of the written user utterance to be a defined intent and/or entity of the intent/entity model that corresponds to the matching written sample utterance within the intent/entity model.

9. The agent automation system of claim 1, wherein the processor is configured to execute instructions of the NLU framework to cause the agent automation system perform additional actions before generating the subtree vectors, comprising:

performing rule-based error detection to detect, in a particular annotated utterance tree of the annotated utterance trees, a misclassification or misparse of words and/or phrases of the corresponding utterance of the particular annotated utterance tree;

performing a rule-based modification of the corresponding utterance of the particular annotated utterance tree to generate a modified written utterance, wherein at least a portion of words and/or phrases of the modified written utterance are different from the words and/or phrases of the corresponding utterance; and

regenerating the particular annotated utterance tree from the modified written utterance using the combination of rules-based and machine-learning (ML)-based components, wherein the regenerated particular annotated utterance tree lacks the misclassification or the misparse of the particular annotated utterance tree before regeneration.

10. The agent automation system of claim 1, wherein the intent/entity model associates defined intents with the written sample utterances, and wherein the written sample utterances encode defined entities as parameters of the defined intents within the intent/entity model.

11. A method of operating a natural language understanding (NLU) framework, comprising:

generating an annotated utterance tree for a written user utterance using a combination of rules-based and machine-learning (ML)-based components, wherein the annotated utterance tree includes nodes arranged in a dependency parse tree structure that represents a syntactic structure of the written user utterance, and wherein each of the nodes includes a word vector representing a semantic meaning of a word or phrase of the written user utterance;

performing rule-based error detection of the annotated utterance tree to detect a misclassification or misparse of words and/or phrases of the written user utterance;

performing a rule-based modification of the written user utterance to generate a modified written utterance, wherein at least a portion of words and/or phrases of the modified written utterance are different from the words and/or phrases of the written user utterance;

regenerating the annotated utterance tree from the modified written utterance using the combination of rules-based and machine-learning (ML)-based components; and

using the regenerated annotated utterance tree as a basis for intent/entity extraction of the written user utterance by;

generating a subtree vector for each subtree of the regenerated annotated utterance tree based on the word vectors of the nodes of each subtree of the regenerated annotated utterance trees; and

extracting an intent and/or entity from the written user utterance based on a comparison of the subtree vectors of the regenerated annotated utterance tree to subtree vectors of other annotated utterance trees representing sample utterances of an intent/entity model.

12. The method of claim 11, comprising performing rules-based cleansing and augmentation to modify the written user utterance before generating the annotated utterance tree.

13. The method of claim 11, wherein generating the annotated utterance tree comprises:

using a prosody subsystem of the NLU framework to analyze the written user utterance for prosody cues to segment the written user utterance;

using a structure subsystem of the NLU framework to analyze the segmented written user utterance to generate and annotate the annotated utterance tree; and

using a vocabulary subsystem of the NLU framework to analyze the written user utterance to provide word vectors for the nodes of the annotated tree based on a word vector distribution model.

14. The method of claim 13, comprising improving operation of the structure subsystem by:

parsing the written user utterance using one or more rules-based parsers of the structure subsystem to generate a plurality of utterance trees; and

in response to determining that a majority of the plurality of utterance trees have a common structure, adjusting a weight of a neural network model associated with a ML-based parser such that the ML-based parser is configured to generate the common structure from the utterance.

15. The method of claim 13, comprising improving operation of the vocabulary subsystem by:

performing rule-based unsupervised learning of words of the written user utterance; and

modifying word vectors of the word vector distribution model based on the rule-based unsupervised learning of the words of the written user utterance.

16. A non-transitory, computer-readable medium storing instructions of a natural language understanding (NLU) framework executable by one or more processors of a computing system, the instructions comprising:

instructions to use a prosody subsystem of the NLU framework to analyze a written user utterance for prosody cues to segment the written user utterance;

instructions to use a structure subsystem of the NLU framework to analyze the segmented utterance to generate an annotated utterance tree having nodes representing words and/or phrases of the written user utterance that are annotated with class information;

instructions to use a vocabulary subsystem of the NLU framework to analyze the utterance to produce word vectors for the nodes of the annotated utterance tree representing the words and/or phrases of the written user utterance; and

instructions to use the annotated utterance tree as a basis for intent/entity extraction of the written user utterance comprising;

instructions to generate a subtree vector for each subtree of the annotated utterance tree based on the word vectors of the nodes of each subtree of the annotated utterance tree; and

instructions to extract an intent and/or entity from the written user utterance based on a comparison of the subtree vectors of the annotated utterance trees of the written user utterance to subtree vectors of other annotated utterance trees of written sample utterances of an intent/entity model.

17. The medium of claim 16, comprising:

instructions to perform rule-based error detection of the annotated utterance tree to detect a misclassification or misparse of words and/or phrases of the written user utterance;

instructions to perform a rule-based modification of the written user utterance to generate a modified written utterance, wherein at least a portion of words and/or phrases of the modified written utterance are different from the words and/or phrases of the written user utterance; and

instructions to regenerate the annotated utterance tree from the modified written utterance using the combination of rules-based and machine-learning (ML)-based components before using the annotated utterance tree as a basis for intent/entity extraction.

18. The medium of claim 16, comprising instructions to improve operation of the structure subsystem and the vocabulary subsystem, including:

instructions to parse the written user utterance using one or more rules-based parsers of the structure subsystem to generate a plurality of utterance trees, and in response a majority of the plurality of utterance trees having a common structure, instructions to adjust a weight of a neural network model associated with a ML-based parser of the structure subsystem based on the common structure; and

instructions to perform rule-based unsupervised learning of the words of the written user utterance and modify word vectors of a word vector distribution model of the vocabulary subsystem based on the rule-based unsupervised learning of the words of the written user utterance.

19. The medium of claim 16, comprising:

instructions to use the prosody subsystem of the NLU framework to analyze the written sample utterances of the intent/entity model for prosody cues to segment each of the written sample utterances;

instructions to use the structure subsystem of the NLU framework to analyze the segmented written sample utterances to generate the other annotated utterance trees having nodes representing words and/or phrases of the written sample utterances, wherein the nodes are annotated with class information; and

instructions to use the vocabulary subsystem of the NLU framework to analyze the written sample utterances to produce word vectors for the nodes of the other annotated utterance trees representing the words and/or phrases of the written sample utterances.