Transferring information across language understanding model domains

• US 10,191,999 B2
• Filed: 04/30/2014
• Issued: 01/29/2019
• Est. Priority Date: 04/30/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Liu, Xiaohu, Elkahky, Ali Mamdouh, Sarikaya, Ruhi, Tur, Gokhan, Hakkani-Tur, Dilek, Heck, Larry Paul
• Primary Examiner(s): Spooner, Lamont M
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US14/266,253
• Publication Number:
  US 20150317302A1
• Time in Patent Office: 1,735 Days
• Field of Search: 704 9, 707736
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding

Abstract

Aspects of the present invention provide a technique to validate the transfer of intents or entities between existing natural language model domains (hereafter “domain” or “NLU”) using click logs, a knowledge graph, or both. At least two different types of transfers are possible. Intents from a first domain may be transferred to a second domain. Alternatively or additionally, entities from the second domain may be transferred to an existing intent in the first domain. Either way, additional intent/entity pairs can be generated and validated. Before the new intent/entity pair is added to a domain, aspects of the present invention validate that the intent or entity is transferable between domains. Validation techniques that are consistent with aspects of the invention can use a knowledge graph, search query click logs, or both to validate a transfer of intents or entities from one domain to another.

Claims

1. A computer-implemented method of expanding slot coverage for a domain-specific natural language understanding (“

NLU”

) system, the method comprising;

accessing a plurality of queries from labeled training data for a classifier used to recognize an intent within the domain-specific NLU, wherein the intent is associated with a slot of the domain-specific NLU;

identifying a plurality of entities that occur in the slot of the domain-specific NLU within the plurality of queries;

extracting, from a knowledge graph, a graph type for each of the plurality of entities to generate a plurality of candidate graph types for the slot of the domain-specific NLU, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;

calculating a correlation score for each graph type in the plurality of candidate graph types for pairing the at least one compatible entity with the recognized intent of the domain-specific NLU;

assigning an individual graph type having the highest correlation score as the graph type the domain-specific NLU slot can accept;

expanding slot coverage for the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned individual graph type with the recognized intent of the domain-specific NLU; and

utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.

2. The method of claim 1, wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.

3. The method of claim 1, wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.

4. The method of claim 1, wherein the correlation score is calculated using a Term Frequency/ Inverse document frequency (“

TF/IDF”

) equation that uses graph types as terms and slots as documents.

5. The method of claim 4, wherein IDF is calculated to avoid down weighting a graph type that appears with many similar slots by determining a similarity between slots.

6. The method of claim 4, wherein TF is calculated using a function that gives a weight that is inversely proportional to a number of graph types appearing with each entity in the knowledge graph.

7. The method of claim 1, wherein the correlation score is calculated using a weighted summation.

8. The method of claim 1, wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“

NLU”

) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.

9. One or more computer-storage media having computer-executable instructions embodied thereon that when executed by a computing device perform the method of expanding slot coverage for a domain-specific natural language understanding (“

NLU”

) system, the method comprising;

accessing a plurality of queries from labeled training data for a classifier used to recognize an intent within the domain-specific NLU, wherein the intent is associated with a slot of the domain-specific NLU;

identifying a plurality of entities that occur in the slot of the domain-specific NLU within the plurality of queries;

extracting, from a knowledge graph, a graph type for each of the plurality of entities to generate a plurality of candidate graph types for the slot of the domain-specific NLU, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;

calculating a correlation score for each graph type in the plurality of candidate graph types for pairing the at least one compatible entity with the recognized intent of the domain-specific NLU;

assigning an individual graph type having the highest correlation score as the graph type the domain-specific NLU slot can accept;

expanding slot coverage for the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned individual graph type with the recognized intent of the domain-specific NLU; and

utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.

10. The method of claim 9, the method further comprising associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.

11. The method of claim 9, wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.

12. The method of claim 9, wherein the correlation score is calculated using a Term Frequency/Inverse document frequency (“

TF/IDF”

) equation that uses graph types as terms and slots as documents.

13. The method of claim 12, wherein IDF is calculated to avoid down weighting a graph type that appears with many similar slots by determining a similarity between slots.

14. The method of claim 12, wherein TF is calculated using a function that gives a weight that is inversely proportional to a number of graph types appearing with each entity in the knowledge graph.

15. The method of claim 9, wherein the correlation score is calculated using a weighted summation.

16. The method of claim 9, wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“

NLU”

) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.

17. A computer-implemented method of expanding slot coverage for a domain-specific natural language understanding (“

NLU”

) system, the method comprising;

accessing a plurality of queries from training data to recognize an intent and a plurality of domain-specific entities, wherein the recognized intent and the plurality of domain-specific entities are associated with a slot of the domain-specific NLU;

generating a plurality of candidate graph types for the slot of the domain-specific NLU by extracting, from a knowledge graph, a plurality of candidate graph types associated with each of the plurality of domain-specific entities, the candidate graph types comprising at least one compatible entity that is eligible for pairing with the recognized intent of the domain-specific NLU;

assigning at least one of the plurality of candidate graph types as an individual graph type that the domain-specific NLU can accept based on a correlation score for each of the plurality of candidate graph types associated with the domain-specific entity, the individual graph type having the highest correlation score;

expanding slot coverage of the domain-specific NLU by validating the pairing of the at least one compatible entity associated with the assigned graph type with the recognized intent of the domain-specific NLU; and

utilizing the validated pairing of the compatible entity and the recognized intent to interpret a natural language input of a query.

18. The method of claim 17, wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type is a child within the knowledge graph of the individual graph type.

19. The method of claim 17, wherein the method further comprises associating a new graph type with the domain-specific NLU slot upon determining the new graph type has an attribute within the knowledge graph of the individual graph type.

20. The method of claim 18, wherein the correlation score is calculated using a weighted summation.

21. The method of claim 18, wherein the method further comprises transferring an entity list from a second domain-specific natural language understanding (“

NLU”

) system upon determining entities in the entity list are associated with a graph type that is compatible with the individual graph type.