Semantic re-ranking of NLU results in conversational dialogue applications

• US 9,761,225 B2
• Filed: 06/25/2014
• Issued: 09/12/2017
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Gandrabur, Simona, Lavallee, Jean-Francois, Tremblay, Real
• Primary Examiner(s): Hang, Vu B
• Application Number:
  US14/314,248
• Publication Number:
  US 20140309990A1
• Time in Patent Office: 1,175 Days
• Field of Search: 704 9, 704257
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Multiple natural language understanding (NLU) interpretation selection models may be generated. The NLU interpretation selection models may include a generic NLU interpretation selection model that is not specialized for a specific set of NLU interpretations type and one or more specialized NLU interpretation selection models, each of which may be specific to a particular set of NLU interpretations type. The specialized NLU interpretation selection model(s) may be utilized to process natural language input data comprising data corresponding to their respective sets of NLU interpretations type(s). The generic NLU interpretation selection model may be utilized to process natural language input data comprising data that does not correspond to the sets of NLU interpretations type(s) associated with the specialized NLU interpretation selection model(s).

Claims

1. A method, comprising:

generating, by at least one processor of a computing platform comprising the at least one processor and a memory, a plurality of natural language understanding (NLU) interpretation selection models that includes a generic NLU interpretation selection model that is not specialized for a specific set of NLU interpretations type, a first specialized NLU interpretation selection model specific to a first set of NLU interpretations type, and a second specialized NLU interpretation selection model, the second specialized NLU interpretation selection model being specific to a second set of NLU interpretations type, the second set of NLU interpretations type being different from the first set of NLU interpretations type;

receiving natural language input data comprising;

a first data portion comprising sets of NLU interpretations types corresponding to the first set of NLU interpretations type,

a second data portion comprising sets of NLU interpretations types corresponding to the second set of NLU interpretations type, and

a third data portion;

parsing the natural language input data to identify the first data portion and the second data portion;

identifying a remainder of the natural language input data as the third data portion;

processing, by the at least one processor, the first data portion using the first specialized NLU interpretation selection model;

processing, by the at least one processor, the second data portion using the second specialized NLU interpretation selection model; and

processing, by the at least one processor, the third data portion using the generic NLU interpretation selection model.

2. The method of claim 1, comprising:

extracting, by the at least one processor, a plurality of sets of NLU interpretations types from a dataset comprising natural language input data; and

classifying, by that least one processor, each set of NLU interpretations type of the plurality of sets of NLU interpretations types as corresponding to a set of NLU interpretations types.

3. The method of claim 2, comprising:

identifying, by the at least one processor and from amongst the plurality of sets of NLU interpretations types, a group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type;

identifying, by the at least one processor and from amongst the plurality of sets of NLU interpretations types, a group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type; and

identifying, by the at least one processor and from amongst the plurality of sets of NLU interpretations types, multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type.

4. The method of claim 3, comprising:

determining, by the at least one processor and based on a number of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, to generate the first specialized NLU interpretation selection model; and

determining, by the at least one processor and based on a number of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, to generate the second specialized NLU interpretation selection model.

5. The method of claim 4, comprising:

responsive to determining to generate the first specialized NLU interpretation selection model, generating the first specialized NLU interpretation selection model, wherein generating the first specialized NLU interpretation selection model comprises executing, by the at least one processor, a machine learning algorithm on a dataset comprising natural language input data that includes the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, and does not include the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type; and

responsive to determining to generate the second specialized NLU interpretation selection model, generating the second specialized NLU interpretation selection model, wherein generating the second specialized NLU interpretation selection model comprises executing, by the at least one processor, the machine learning algorithm on a dataset comprising natural language input data that includes the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type.

6. The method of claim 3, comprising:

determining, by the at least one processor, for each set of NLU interpretations type of the sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, and based on a number of sets of NLU interpretations types classified as corresponding to the set of NLU interpretations type, not to generate a specialized NLU interpretation selection model specific to the set of NLU interpretations type; and

for each set of NLU interpretations type of the sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, responsive to determining not to generate the specialized NLU interpretation selection model specific to the set of NLU interpretations type, adding each NLU interpretations type corresponding to the set of NLU interpretations type to a common dataset to form a dataset comprising natural language input data that includes the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type.

7. The method of claim 6, comprising generating the generic NLU interpretation selection model, wherein generating the generic NLU interpretation selection model comprises executing, by the at least one processor, a machine learning algorithm on the dataset comprising natural language input data that includes the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type.

8. The method of claim 1, comprising:

parsing, by the at least one processor, the natural language input data comprising data corresponding to the first set of NLU interpretations type to identify the data corresponding to the first set of NLU interpretations type; and

parsing, by the at least one processor, the natural language input data comprising data corresponding to the second set of NLU interpretations type to identify the data corresponding to the second set of NLU interpretations type.

9. The method of claim 8, comprising:

responsive to identifying the data corresponding to the first set of NLU interpretations type, identifying, by the at least one processor, the first specialized NLU interpretation selection model for utilization, by the at least one processor, to process the natural language input data comprising the data corresponding to the first set of NLU interpretations type; and

responsive to identifying the data corresponding to the second set of NLU interpretations type, identifying, by the at least one processor, the second specialized NLU interpretation selection model for utilization, by the at least one processor, to process the natural language input data comprising the data corresponding to the second set of NLU interpretations type.

10. The method of claim 1, comprising:

parsing, by the at least one processor, the natural language input data comprising data corresponding to neither of the first set of NLU interpretations type and the second set of NLU interpretations type to identify the data corresponding to neither of the first set of NLU interpretations type and the second set of NLU interpretations type; and

responsive to identifying the data corresponding to neither of the first set of NLU interpretations type and the second set of NLU interpretations type, identifying, by the at least one processor, the generic NLU interpretation selection model for utilization, by the at least one processor, to process the natural language input data comprising data corresponding to neither of the first set of NLU interpretations type and the second set of NLU interpretations type.

11. An apparatus, comprising:

at least one processor; and

a memory storing instructions that when executed by the at least one processor cause the apparatus to;

generate a generic NLU interpretation selection model that is not specialized for a specific set of NLU interpretations type;

generate a first specialized NLU interpretation selection model specific to a first set of NLU interpretations type;

generate a second specialized NLU interpretation selection model, the second specialized NLU interpretation selection model being specific to a second set of NLU interpretations type, the second set of NLU interpretations type being different from the first set of NLU interpretations type;

receive natural language input data comprising;

a first data portion comprising sets of NLU interpretations types corresponding to the first set of NLU interpretations type,

a second data portion comprising sets of NLU interpretations types corresponding to the second set of NLU interpretations type, and

a third data portion;

parse the natural language input data to identify the first data portion and the second data portion;

identify a remainder of the natural language input data as the third data portion;

process the first data portion using the first specialized NLU interpretation selection model;

process the second data portion using the second specialized NLU interpretation selection model; and

process the third data portion using the generic NLU interpretation selection model.

12. The apparatus of claim 11, wherein the instructions, when executed by the at least one processor, cause the apparatus to:

extract a plurality of sets of NLU interpretations types from a dataset comprising natural language input data;

classify each set of NLU interpretations types of the plurality of sets of NLU interpretations types as corresponding to a set of NLU interpretations type;

identify, from amongst the plurality of sets of NLU interpretations types, a group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type;

identify, from amongst the plurality of sets of NLU interpretations types, a group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type; and

identify, from amongst the plurality of sets of NLU interpretations types, multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type.

13. The apparatus of claim 12, wherein the instructions, when executed by the at least one processor, cause the apparatus to:

determine, based on a number of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, to generate the first specialized NLU interpretation selection model;

determine, based on a number of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, to generate the second specialized NLU interpretation selection model;

responsive to determining to generate the first specialized NLU interpretation selection model, generate the first specialized NLU interpretation selection model by executing a machine learning algorithm on a dataset comprising natural language input data that includes the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, and does not include the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type; and

responsive to determining to generate the second specialized NLU interpretation selection model, generate the second specialized NLU interpretation selection model by executing the machine learning algorithm on a dataset comprising natural language input data that includes the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type.

14. The apparatus of claim 12, wherein the instructions, when executed by the at least one processor, cause the apparatus to:

determine, for each set of NLU interpretations type of the sets of NLU interpretations types different from both the first NLU interpretations type and the second NLU interpretations type, and based on a number of sets of NLU interpretations types classified as corresponding to the set of NLU interpretations type, not to generate a specialized NLU interpretation selection model specific to the set of NLU interpretations type;

for each set of NLU interpretations type of the sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, responsive to determining not to generate the specialized NLU interpretation selection model specific to the set of NLU interpretations type, add each NLU interpretations type corresponding to the set of NLU interpretations type to a common dataset to form a dataset comprising natural language input data that includes the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type; and

generate the generic NLU interpretation selection model by executing a machine learning algorithm on the dataset comprising natural language input data that includes the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type.

15. The apparatus of claim 11, wherein the instructions, when executed by the at least one processor, cause the apparatus to:

parse the natural language input data comprising data corresponding to the first set of NLU interpretations type to identify the data corresponding to the first set of NLU interpretations type;

parse the natural language input data comprising data corresponding to the second set of NLU interpretations type to identify the data corresponding to the second set of NLU interpretations type;

responsive to identifying the data corresponding to the first set of NLU interpretations type, identify the first specialized NLU interpretation selection model for utilization, by the at least one processor, to process the natural language input data comprising the data corresponding to the first set of NLU interpretations type; and

responsive to identifying the data corresponding to the second set of NLU interpretations type, identify the second specialized NLU interpretation selection model for utilization, by the at least one processor, to process the natural language input data comprising the data corresponding to the second set of NLU interpretations type.

16. One or more non-transitory computer-readable media having instructions stored thereon that when executed by one or more computers cause the one or more computers to:

generate a generic NLU interpretation selection model that is not specialized for a specific set of NLU interpretations type;

generate a first specialized NLU interpretation selection model specific to a first set of NLU interpretations type;

generate a second specialized NLU interpretation selection model, the second specialized NLU interpretation selection model being specific to a second set of NLU interpretations type, the second set of NLU interpretations type being different from the first set of NLU interpretations type;

receive natural language input data comprising;

a first data portion comprising sets of NLU interpretations types corresponding to the first set of NLU interpretations type,

a second data portion comprising sets of NLU interpretations types corresponding to the second set of NLU interpretations type, and

a third data portion;

parse the natural language input data to identify the first data portion and the second data portion;

identify a remainder of the natural language input data as the third data portion;

process the first data portion using the first specialized NLU interpretation selection model;

process the second data portion using the second specialized NLU interpretation selection model; and

process the third data portion using the generic NLU interpretation selection model.

17. The one or more non-transitory computer-readable media of claim 16, wherein the instructions, when executed by the one or more computers, cause the one or more computers to:

extract a plurality of sets of NLU interpretations types from a dataset comprising natural language input data;

classify each set of NLU interpretations types of the plurality of sets of NLU interpretations types as corresponding to a set of NLU interpretations type;

identify, from amongst the plurality of sets of NLU interpretations types, a group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type;

identify, from amongst the plurality of sets of NLU interpretations types, a group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type; and

identify, from amongst the plurality of sets of NLU interpretations types, multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type.

18. The one or more non-transitory computer-readable media of claim 17, wherein the instructions, when executed by the one or more computers, cause the one or more computers to:

determine, based on a number of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, to generate the first specialized NLU interpretation selection model;

determine, based on a number of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, to generate the second specialized NLU interpretation selection model;

responsive to determining to generate the first specialized NLU interpretation selection model, generate the first specialized NLU interpretation selection model by executing a machine learning algorithm on a dataset comprising natural language input data that includes the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type, and does not include the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type; and

responsive to determining to generate the second specialized NLU interpretation selection model, generate the second specialized NLU interpretation selection model by executing the machine learning algorithm on a dataset comprising natural language input data that includes the group of sets of NLU interpretations types classified as corresponding to the second NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type.

19. The one or more non-transitory computer-readable media of claim 17, wherein the instructions, when executed by the one or more computers, cause the one or more computers to:

determine, for each set of NLU interpretations type of the sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, and based on a number of sets of NLU interpretations types classified as corresponding to the set of NLU interpretations type, not to generate a specialized NLU interpretation selection model specific to the set of NLU interpretations type;

for each set of NLU interpretations type of the set of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, responsive to determining not to generate the specialized NLU interpretation selection model specific to the set of NLU interpretations type, add each NLU interpretations type corresponding to the set of NLU interpretations type to a common dataset to form a dataset comprising natural language input data that includes the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type; and

generate the generic NLU interpretation selection model by executing a machine learning algorithm on the dataset comprising natural language input data that includes the multiple groups of sets of NLU interpretations types classified as corresponding to sets of NLU interpretations types different from both the first set of NLU interpretations type and the second set of NLU interpretations type, does not include the group of sets of NLU interpretations types classified as corresponding to the first set of NLU interpretations type, and does not include the group of sets of NLU interpretations types classified as corresponding to the second set of NLU interpretations type.

20. The one or more non-transitory computer-readable media of claim 16, wherein the instructions, when executed by the one or more computers, cause the one or more computers to:

parse the natural language input data comprising data corresponding to the first set of NLU interpretations type to identify the data corresponding to the first set of NLU interpretations type;

parse the natural language input data comprising data corresponding to the second set of NLU interpretations type to identify the data corresponding to the second set of NLU interpretations type;

responsive to identifying the data corresponding to the first set of NLU interpretations type, identify the first specialized NLU interpretation selection model for utilization, by the one or more computers, to process the natural language input data comprising the data corresponding to the first set of NLU interpretations type; and

responsive to identifying the data corresponding to the second set of NLU interpretations type, identify the second specialized NLU interpretation selection model for utilization, by the one or more computers, to process the natural language input data comprising the data corresponding to the second set of NLU interpretations type.