Generating and using a knowledge-enhanced model

• US 10,089,580 B2
• Filed: 08/11/2014
• Issued: 10/02/2018
• Est. Priority Date: 08/11/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shan, Ying, Mao, Jianchang
• Primary Examiner(s): Wong, Lut
• Application Number:
  US14/456,985
• Publication Number:
  US 20160042296A1
• Time in Patent Office: 1,513 Days
• Field of Search: 706 20
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Functionality is described herein for generating a model on the basis of user-behavioral data and knowledge data. In one case, the user-behavioral data identifies queries submitted by users, together with selections made by the users in response to the queries. The knowledge data represents relationships among linguistic items, as expressed by one or more structured knowledge resources. The functionality leverages the knowledge data to supply information regarding semantic relationships which may not be adequately captured by the user-behavioral data, to thereby produce a more robust and accurate model (compared to a model produced on the basis of only user-behavioral data). Functionality is also described herein for applying the model, once trained. In one case, the model may correspond to a deep learning model.

Claims

1. A method implemented by one or more computing devices, the method comprising:

sampling first click-through data from a repository, the first click-through data identifying queries submitted by users to a search engine and specific result items that the users clicked from search results provided by the search engine in response to the queries;

sampling structured knowledge data from one or more structured knowledge resources, the structured knowledge data providing semantic distances between various nouns identified in the one or more structured knowledge resources;

processing the structured knowledge data to obtain second click-through the data second click-through data representing respective semantic distances between semantically related nouns as corresponding click values; and

training a model using the first click-through data and the second click-through data as training data, the model being trained using a machine-learning training process,

wherein the model is configured to process input linguistic items and identify output linguistic items that are related to the input linguistic items.

2. The method of claim 1, wherein each instance of the first click-through data represents at least a particular query, an answer to the particular query, and an associated click value that indicates whether a user selected the answer in response to the particular query.

3. The method of claim 1, wherein each instance of the structured knowledge data comprises at least one frequency measure associated with a pair of semantically related nouns.

4. The method of claim 1, wherein the structured knowledge data includes:

a first subset of instances that represent pairs of semantically related nouns, and

a second subset of instances that represent pairs of unrelated nouns.

5. The method of claim 1, wherein the click values that represent the respective semantic distances between the semantically related nouns are positive numerical values.

6. The method of claim 5, the positive numerical values being one, wherein the click values represent respective semantic distances between unrelated nouns as zero.

7. The method of claim 1, wherein the model is configured to map the input linguistic items and the output linguistic items into a semantic space.

8. The method of claim 7, wherein the model comprises a neural network.

9. The method of claim 8, wherein training the neural network comprises:

iteratively adjusting neural network parameters according to a result of an objective function.

10. The method of claim 9, wherein the objective function maximizes a conditional likelihood of clicked linguistic items given respective input linguistic items and minimizes a conditional likelihood of non-clicked linguistic items given the respective input linguistic items.

11. A computer readable storage medium storing computer readable instructions, the computer readable instructions providing a semantic transformation module when executed by one or more processing devices, the computer readable instructions comprising:

logic configured to;

use a deep learning model to map an input linguistic item into a concept vector in a high-level conceptual space; and

identify, in the high-level conceptual space, one or more output linguistic items that are related to the input linguistic item,

the deep learning model capturing semantic relationships learned in a machine-learning training process performed on training instances of user-behavioral data and other training instances of structured knowledge data,

the training instances of user-behavioral data identifying user-submitted linguistic items submitted by users to a search engine together with user clicks made by the users on user-clicked result items provided by the search engine in response to the user-submitted linguistic items, and

the other training instances of structured knowledge data representing semantic distances between nouns expressed by one or more structured knowledge resources as corresponding click values.

12. A computer system, comprising:

a processing device; and

a storage resource storing instructions which, when executed by the processing device, cause the processing device to implement;

a search engine configured to receive an input linguistic item, and configured to identify at least one output item that has been determined to be relevant to the input linguistic item,

the search engine being configured to identify said at least one output item using a model configured to map the input linguistic item and the at least one output item into a semantic space,

the model being trained by a machine-learning training process based at least on user-behavioral training data and structured knowledge training data,

the user-behavioral training data identifying user-submitted linguistic items submitted by users together with user clicks made by the users on specific result items provided by a search engine in response to the user-submitted linguistic items, and

the structured knowledge training data representing, as corresponding click values, semantic distances between semantically-related nouns, expressed by one or more structured knowledge resources.

13. The computer system of claim 12, wherein the instructions, when executed by the processing device, cause the processing device to:

train the model based at least on the user-behavioral training data and the structured knowledge training data.

14. The computer system of claim 13, wherein the instructions, when executed by the processing device, cause the processing device to:

access structured knowledge data in the one or more structured knowledge resources, the structured knowledge data identifying the semantic distances; and

perform a conversion of the structured knowledge data to obtain the structured knowledge training data, the conversion involving associated individual click values with pairs of nouns identified in the structured knowledge data.

15. The computer system of claim 14, wherein the user-behavioral training data comprises multiple instances, and each instance of the user-behavioral training data represents at least a particular query, a particular answer to the particular query, and a corresponding click value that indicates whether a user selected the particular answer in response to the particular query.

16. The computer system of claim 14, wherein the structured knowledge training data comprises multiple instances, and each instance of the structured knowledge training data represents at least a pair of nouns, a corresponding click value that represents a degree of semantic relatedness between the pair of nouns as identified by the one or more structured knowledge resources, and at least one frequency measure associated with the pair of nouns.

17. The computer system of claim 14, wherein the model is configured to determine relevance measures reflecting relevance of multiple output items to the input linguistic item.

18. The computer system of claim 17, wherein the instructions, when executed by the processing device, cause the processing device to:

rank individual output items based at least on the relevance measures.

19. The computer system of claim 12, wherein the model corresponds to a deep learning model that uses a multi-layer neural network to project the input linguistic item into a concept vector expressed in a semantic space.

20. The computer system of claim 12, wherein the model corresponds to any type of model other than a deep learning model.