Generation and application of universal hypothesis ranking model

• US 10,339,916 B2
• Filed: 08/31/2015
• Issued: 07/02/2019
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sarikaya, Ruhi, Crook, Paul Anthony
• Primary Examiner(s): Shah, Bharatkumar S
• Application Number:
  US14/841,156
• Publication Number:
  US 20170061956A1
• Time in Patent Office: 1,401 Days
• Field of Search: 704205
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning

Abstract

Non-limiting examples of the present disclosure describe generation and application of a universal hypothesis ranking model to rank/re-re-rank dialog hypotheses. An input is received through a user interface of an application for dialog processing. A plurality of dialog hypotheses are generated based on input understanding processing of the received input. The plurality of dialog hypotheses are ranked using a universal hypothesis ranking model that is applicable to a plurality of languages and locales. The ranking of the plurality of dialog hypotheses comprises using the universal hypothesis ranking model to analyze language independent features of the plurality of dialog hypotheses for policy determination. Other examples are also described including examples directed to generation of the universal hypothesis ranking model.

Claims

1. A system comprising:

at least one processor; and

a memory operatively connected with the processor, wherein the memory stores computer-executable instructions, that cause the processor to perform;

analyzing a corpus of training data, wherein the corpus of training data comprises data received in a plurality of different languages;

creating, based on the analyzed corpus of training data, a language-independent feature set that includes selectable features for ranking of dialog hypotheses; and

training a single model using the language-independent feature set to generate a universal hypothesis ranking model, wherein the universal hypothesis ranking model is applicable to a plurality of languages and locales, and wherein the universal hypothesis ranking model is configured to rank hypotheses for user input received in a language previously unseen by the universal hypothesis ranking model.

2. The system according to claim 1, wherein the data for the plurality of different languages is obtained from two or more language-specific dialog application components stored in a memory of one or more processing devices.

3. The system according to claim 1, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform storing the universal hypothesis ranking model in a memory of at least one processing device.

4. The system according to claim 1, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform providing the generated universal hypothesis ranking model as at least one from a group consisting of:

a file, a distributable package, and an application, for distributed access to a plurality of processing devices in a distributed network, and wherein the plurality of processing devices execute at least one application for dialog processing.

5. The system according to claim 4, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform:

receiving, over the distributed network from another processing device, a received input from the application for dialog processing,

generating, a plurality of dialog hypotheses in response to performing language understanding processing on the received input,

ranking the plurality of dialog hypotheses using the universal hypothesis ranking model, and

transmitting one or more of the ranked dialog hypotheses to the another processing device over the distributed network.

6. The system according to claim 5, wherein the ranking further comprises extracting a selectable feature set comprising two or more features from the language-independent feature set, and analyzing a dialog hypothesis of the plurality of dialog hypotheses using the extracted selectable feature set.

7. The system according to claim 1, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform:

integrating the universal hypothesis ranking model into at least one application for dialog processing,

storing the at least one application for dialog processing in the memory, and

executing the application for dialog processing, wherein the executing further comprises;

displaying, on a display of a processing device, a graphical user interface, and

presenting, through the graphical user interface, natural language phrases that are displayed on the processing device and spoken to a user.

8. The system according to claim 7, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform:

receiving, through the user interface, an input,

generating, using the application for dialog processing, a plurality of dialog hypotheses in response to performing language understanding processing on the received input,

ranking the plurality of dialog hypotheses using the universal hypothesis ranking model, and

outputting one or more of the ranked dialog hypotheses for display through the user interface.

9. The system according to claim 8, wherein the ranking further comprises extracting a selectable feature set comprising two or more features from the language-independent feature set, and analyzing a dialog hypothesis of the plurality of dialog hypotheses using the extracted selectable feature set.

10. A computer-implemented method comprising:

receiving an input through a user interface of an application for dialog processing;

generating a plurality of dialog hypotheses based on input understanding processing of the received input; and

ranking the plurality of dialog hypotheses using a universal hypothesis ranking model that is applicable to a plurality of languages and locales, wherein the universal hypothesis ranking model is configured to rank hypotheses for user input received in a language previously unseen by the universal hypothesis ranking model;

wherein the ranking comprises;

applying the universal hypothesis ranking model to analyze language independent features of the plurality of dialog hypotheses extracted by the universal hypothesis ranking model; and

outputting a ranking of the plurality of dialog hypotheses for policy determination.

11. The computer-implemented method according to claim 10, wherein the generating of the dialog hypotheses further comprises executing a knowledge fetch to analyze semantic frame data generated by language understanding processing, and wherein the executed knowledge fetch uses knowledge data obtained from at least one database to analyze the semantic frame data.

12. The computer-implemented method according to claim 10, further comprising executing a policy determination that outputs one or more of the ranked plurality of dialog hypotheses.

13. The computer-implemented method according to claim 12, wherein the ranking further comprises extracting a plurality of language independent features for each of the plurality of dialog hypotheses, and scoring the plurality of dialog hypotheses based on analyzing the extracted features using training data of the universal language model, wherein the training data comprises data obtained from user input received in a plurality of different languages.

14. The computer-implemented method according to claim 12, further comprising:

during processing of the received input, retaining data associated with one or more selected from a group consisting of input recognition processing, input understanding processing and dialog processing,

receiving a subsequent input, and

utilizing the retained data to enhance processing of the subsequent input during at least one selected from group consisting of input recognition processing, input understanding processing and dialog processing.

15. A system comprising:

at least one processor; and

a memory operatively connected with the processor, wherein the memory stores computer-executable instructions, that cause the processor to perform;

receiving an input through a user interface of an application for dialog processing,

generating a plurality of dialog hypotheses based on input understanding processing of the received input, and

ranking the plurality of dialog hypotheses using a universal hypothesis ranking model that is applicable to a plurality of languages and locales, wherein the universal hypothesis ranking model is configured to rank hypotheses for user input received in a language previously unseen by the universal hypothesis ranking model;

wherein the ranking comprises;

applying the universal hypothesis ranking model to analyze language independent features of the plurality of dialog hypotheses extracted by the universal hypothesis ranking model; and

outputting a ranking of the plurality of dialog hypotheses for policy determination.

16. The system according to claim 15, wherein the generating of the dialog hypotheses further comprises executing a knowledge fetch to analyze semantic frame data generated by language understanding processing, wherein the executed knowledge fetch uses knowledge data obtained from at least one database to analyze the semantic frame data.

17. The system according to claim 15, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform executing a policy determination that outputs one or more of the ranked plurality of dialog hypotheses.

18. The system according to claim 17, wherein the ranking further comprises extracting a plurality of language independent features for each of the plurality of dialog hypotheses, and scoring the plurality of dialog hypotheses based on analyzing the extracted features using training data of the universal hypothesis ranking model, wherein the training data comprises data obtained from user input received in a plurality of different languages.

19. The system according to claim 17, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform:

during processing of the received input, retaining data associated with one or more selected from a group consisting of input recognition processing, input understanding processing and dialog processing.

20. The system according to claim 19, wherein the computer-executable instructions further comprise an instruction that causes the processor to perform:

receiving a subsequent input, and utilizing the retained data to enhance processing of the subsequent input during at least one selected from group consisting of input recognition processing, input understanding processing and dialog processing.