Dimensionally reduction of linguistics information

• US 10,055,686 B2
• Filed: 07/12/2016
• Issued: 08/21/2018
• Est. Priority Date: 09/06/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Huang, Po-Sen, He, Xiaodong, Gao, Jianfeng, Deng, Li, Acero, Alejandro, Heck, Larry P.
• Primary Examiner(s): Smith, Paulinho E
• Application Number:
  US15/207,896
• Publication Number:
  US 20160321321A1
• Time in Patent Office: 770 Days
• Field of Search: None
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 16/36 Creation of semantic tools, e.g. ontology or thesauri
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

A deep structured semantic module (DSSM) is described herein which uses a model that is discriminatively trained based on click-through data, e.g., such that a conditional likelihood of clicked documents, given respective queries, is maximized, and a condition likelihood of non-clicked documents, given the queries, is reduced. In operation, after training is complete, the DSSM maps an input item into an output item expressed in a semantic space, using the trained model. To facilitate training and runtime operation, a dimensionality-reduction module (DRM) can reduce the dimensionality of the input item that is fed to the DSSM. A search engine may use the above-summarized functionality to convert a query and a plurality of documents into the common semantic space, and then determine the similarity between the query and documents in the semantic space. The search engine may then rank the documents based, at least in part, on the similarity measures.

Claims

1. A method performed by a computing device, the method comprising:

obtaining a query comprising one or more words from a vocabulary having a first dimension;

transforming the one or more words of the query into a phonetic representation of the one or more words;

processing the phonetic representation to obtain a lower-dimension representation comprising a plurality of n-grams in an n-gram space having a second dimension that is smaller than the first dimension;

performing a natural language processing operation on the lower-dimension representation, the natural language processing operation comprising determining similarity measures reflecting similarity of the one or more words of the query to a plurality of documents;

based at least on the similarity measures, selecting a subset of the documents that are relevant to the query; and

outputting the selected subset of documents in response to the query.

2. The method of claim 1, further comprising:

receiving the query from a user device over a network; and

in response to the query, sending search results identifying the selected subset of documents to the user device over the network.

3. The method of claim 1, further comprising:

transforming the documents into other phonetic representations; and

processing the other phonetic representations to obtain other lower-dimension representations of the documents,

the natural language processing operation comprising;

mapping the lower-dimension representation of the query and the other lower-dimension representations of the documents into a semantic space; and

determining the similarity measures based at least on relative distances between the lower-dimension representation of the query and other lower-dimension representations of the documents in the semantic space.

4. The method of claim 3, wherein the one or more words are obtained as a bag-of-words representation of the query.

5. The method of claim 3, further comprising:

training a semantic model to map the lower-dimension representation of the query and the other lower-dimension representations of the documents into the semantic space using click-through data reflecting user selections of various documents after submitting various queries.

6. The method of claim 1, wherein the processing comprises adding tokens to the phonetic representation, individual n-grams of the lower-dimension representation comprising individual tokens.

7. A system comprising:

a processing device; and

a computer readable storage medium storing instructions which, when executed by the processing device, cause the processing device to;

obtain a query comprising one or more words from a vocabulary having a first dimension;

transform the one or more words of the query into a phonetic representation of the query;

process the phonetic representation of the query to obtain a lower-dimension representation of the query, the lower-dimension representation comprising a plurality of n-grams in an n-gram space having a second dimension that is smaller than the first dimension;

use the lower-dimension representation to determine similarity measures reflecting similarity of the query to a plurality of documents;

based at least on the similarity measures, select a subset of the documents that are relevant to the query; and

output the selected subset of documents in response to the query.

8. The system of claim 7, wherein the instructions, when executed by the processing device, cause the processing device to:

obtain click-through data reflecting user selections of various documents after submitting various queries;

transform the various documents and the various queries into various phonetic representations;

process the various phonetic representations of the various documents and the various queries to obtain various lower-dimension representations comprising various n-grams in the n-gram space; and

train a semantic model using the click-through data to map the various lower-dimension representations into a semantic space.

9. The system of claim 8, wherein the instructions, when executed by the processing device, cause the processing device to:

transform the documents into other phonetic representations of the documents; and

process the other phonetic representations of the documents to obtain other lower-dimension representations of the documents,

map the lower-dimension representation of the query and the other lower-dimension representations of the documents into the semantic space using the semantic model; and

determine the similarity measures based at least on relative distances between the lower-dimension representation of the query and other lower-dimension representations of the documents in the semantic space.

10. The system of claim 9, wherein the instructions, when executed by the processing device, cause the processing device to:

receive the query from a user device over a network; and

in response to the query, send search results identifying the selected subset of documents to the user device over the network.

11. The system of claim 10, wherein the instructions, when executed by the processing device, cause the processing device to:

train the semantic model by iteratively adjusting model parameters of the semantic model to achieve an objective defined in an objective function.

12. The system of claim 11, wherein the objective function increases a conditional likelihood of clicked documents given respective queries.

13. The system of claim 12, wherein the objective function reduces a conditional likelihood of non-clicked documents given the respective queries.

14. A computer readable storage medium storing computer readable instructions which, when executed by one or more processing devices, cause the one or more processing devices to perform acts comprising:

obtaining a query comprising one or more words from a vocabulary having a first dimension;

transforming the query into a phonetic representation of query;

processing the phonetic representation to obtain a lower-dimension representation of the query, the lower-dimension representation comprising a plurality of n-grams in an n-gram space having a second dimension that is smaller than the first dimension;

using the lower-dimension representation of the query to determine similarity measures reflecting similarity of the query to a plurality of documents;

based at least on the similarity measures, selecting a subset of the documents that are relevant to the query; and

outputting the selected subset of documents in response to the query.

15. The computer readable storage medium of claim 14, the acts further comprising:

transforming the documents into other phonetic representations of the documents;

processing the other phonetic representations to obtain other lower-dimension representations of the documents in the n-gram space; and

using the other phonetic representations of the documents with the lower-dimension representation of the query to determine the similarity measures.

16. The computer readable storage medium of claim 15, the acts further comprising:

comparing the lower-dimension representation of the query directly to the other lower-dimension representations of the documents to determine the similarity measures.

17. The computer readable storage medium of claim 15, the acts further comprising:

adding a token to the beginning of the phonetic representation of the query to obtain the lower-dimension representation of the query; and

adding the token to the beginning of the other phonetic representations of the documents to obtain the other lower-dimension representations of the documents.

18. The computer readable storage medium of claim 17, the acts further comprising:

adding the token to the end of the phonetic representation of the query to obtain the lower-dimension representation of the query; and

adding the token to the end of the other phonetic representations of the documents to obtain the other lower-dimension representations of the documents.

19. The computer readable storage medium of claim 18, the similarity measures being determined using the phonetic representation of the query having the token added to the beginning and the end of the phonetic representation, and the other phonetic representations of the documents having the token added to the beginning and the end of the other phonetic representations.

20. The computer readable storage medium of claim 19, the token being a character other than a letter.