Deep structured semantic model produced using click-through data

• US 9,519,859 B2
• Filed: 09/06/2013
• Issued: 12/13/2016
• Est. Priority Date: 09/06/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Huang, Po-Sen, He, Xiaodong, Gao, Jianfeng, Deng, Li, Acero, Alejandro, Heck, Larry P.
• Primary Examiner(s): Smith, Paulinho E
• Application Number:
  US14/019,563
• Publication Number:
  US 20150074027A1
• Time in Patent Office: 1,194 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A deep structured semantic module (DSSM) is described herein which uses a model that is discriminatively trained based on click-through data, e.g., such that a conditional likelihood of clicked documents, given respective queries, is maximized, and a condition likelihood of non-clicked documents, given the queries, is reduced. In operation, after training is complete, the DSSM maps an input item into an output item expressed in a semantic space, using the trained model. To facilitate training and runtime operation, a dimensionality-reduction module (DRM) can reduce the dimensionality of the input item that is fed to the DSSM. A search engine may use the above-summarized functionality to convert a query and a plurality of documents into the common semantic space, and then determine the similarity between the query and documents in the semantic space. The search engine may then rank the documents based, at least in part, on the similarity measures.

Claims

1. One or more computing devices comprising:

a processing device; and

a projection module executable on the processing device, the projection module comprising;

a dimensionality-reduction module configured to;

receive an input item that represents linguistic information comprising a plurality of input words from a vocabulary space having a first dimensionality; and

transform the input item into a lower-dimension item that represents individual input words in another space having a second dimensionality that is smaller than the first dimensionality of the vocabulary space; and

a deep structured semantic module configured to, after the input item has been transformed into the lower-dimension item;

receive the lower-dimension item from the dimensionality-reduction module; and

project, using a model, the lower-dimension item into an output item other than the lower-dimension item,

the output item being expressed in a semantic space, and

the model being discriminatively trained based on click-through data.

2. The one or more computing devices of claim 1, wherein the model is trained using the click-through data such that a conditional likelihood of clicked documents, given respective queries, is maximized, and the conditional likelihood of un-clicked documents, given the respective queries, is reduced.

3. The one or more computing devices of claim 1, wherein each instance of click-through data comprises a set that includes:

a query;

a document which has been selected in response to submission of the query; and

at least one other document that is assumed to have not been selected in response to submission of the query.

4. The one or more computing devices of claim 1, wherein the deep structured semantic module comprises a deep neural network having at least one hidden layer.

5. The one or more computing devices of claim 4, wherein the model is expressed by a plurality of weighting matrices, each weighting matrix providing weighting values for use in projecting values associated with a first layer into values associated with a second layer, the first layer being coupled to the second layer.

6. The one or more computing devices of claim 1, wherein the dimensionality-reduction module is configured to transform the input item using a random projection technique.

7. The one or more computing devices of claim 1, wherein the dimensionality-reduction module is configured to transform the input item by:

expressing the input item as a plurality of n-grams; and

forming the lower-dimension item based on the plurality of n-grams,

each n-gram corresponding to a sequence of letters associated with the input item.

8. The one or more computing devices of claim 1, wherein the dimensionality-reduction module is configured to perform the transforming by:

converting the input item into a converted item;

expressing the converted item as a plurality of n-grams; and

forming the lower-dimension item based on the plurality of n-grams,

each n-gram corresponding to a sequence of letters associated with the converted item.

9. The one or more computing devices of claim 8, wherein the dimensionality-reduction module is configured to perform said converting by providing a phonetic representation of the input item.

10. A method implemented by one or more computing devices the method comprising:

receiving a first input item which represents first linguistic information provided in a vocabulary space having a first dimensionality;

converting the first input item into a phonetic representation of the first input item;

projecting, using a model, the phonetic representation of the first input item into a first output item, the first output item being expressed in a semantic space;

receiving a second output item which represents second linguistic information, the second output item also being expressed in the semantic space; and

determining a similarity between the first output item and the second output item in the semantic space to obtain a similarity measure between the first linguistic information and the second linguistic information,

said projecting using a deep neural network, and

the model being discriminatively trained based on click-through data.

11. The method of claim 10, wherein the first linguistic information is a query, and the second linguistic information is associated with a document.

12. The method of claim 10, wherein the second output item is also produced using a deep neural network, using a model trained based on the click-through data.

13. The method of claim 10, wherein the model is trained using the click-through data such that, for a given query in the click-through data, relevant documents are distinguished from less relevant documents.

14. The method of claim 10, wherein each instance of click-through data comprises a set that includes:

a query;

a document which has been selected in response to submission of the query; and

at least one other document that is assumed to have not been selected in response to submission of the query.

15. The method of claim 10, further comprising:

converting the second linguistic information into a phonetic representation of the second linguistic information; and

using the deep neural network to project the phonetic representation of the second linguistic information into the second output item.

16. A system comprising:

a processing device; and

a computer readable storage medium storing instructions which, when executed by the processing device, cause the processing device to;

receive an input item that represents linguistic information comprising an input word from a vocabulary space having a first dimensionality;

represent the input item as a plurality of n-grams;

map the input into a lower-dimension item that represents the plurality of n-grams in another space having a second dimensionality that is smaller than the first dimensionality of the vocabulary space of the linguistic information; and

use a model to project the lower-dimension item that represents the plurality of n-grams into a semantic space to obtain a semantic output item representing the input item,

wherein the model is trained using click-through data.

17. The system of claim 16, wherein the instructions, when executed, further cause the processing device to:

represent documents retrieved via various queries from the click-through data and represent the documents as other n-grams;

map the other n-grams representing the documents into other lower-dimension items representing the other n-grams in the another space;

use the model to project the other lower-dimension items into the semantic space to obtain other semantic output items representing the documents; and

determine similarities of the input item to the documents by comparing the semantic output item representing the input item to the other semantic output items representing the documents.

18. The system of claim 17, wherein the click-through data represents instances where various users have selected individual documents after submitting individual queries.

19. The system of claim 17, wherein the semantic output item representing the input item comprises a vector and the other semantic output items representing the documents comprise other vectors.

20. The system of claim 17, wherein the vocabulary space of the input item comprises approximately 500,000 possible words and the another space of the lower-dimension item comprises approximately 30,000 accepted n-grams.