Convolutional Latent Semantic Models and their Applications

• US 20150278200A1
• Filed: 04/01/2014
• Published: 10/01/2015
• Est. Priority Date: 04/01/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Xiaodong, Gao, Jianfeng, Deng, Li, Zhou, Yunhong, Buehrer, Gregory T., Mao, Jianchang, Zhang, Ruofei, Lou, Qiang, Liu, Guowei, Shen, Yelong
• Granted Patent: US 9,477,654 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/194 Calculation of difference between files
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Functionality is described herein for transforming first and second symbolic linguistic items into respective first and second continuous-valued concept vectors, using a deep learning model, such as a convolutional latent semantic model. The model is designed to capture both the local and global linguistic contexts of the linguistic items. The functionality then compares the first concept vector with the second concept vector to produce a similarity measure. More specifically, the similarity measure expresses the closeness between the first and second linguistic items in a high-level semantic space. In one case, the first linguistic item corresponds to a query, and the second linguistic item may correspond to a phrase, or a document, or a keyword, or an ad, etc. In one implementation, the convolutional latent semantic model is produced in a training phase based on click-through data.

Claims

1. A method, implemented by one or more computing devices, for processing linguistic items, comprising:

receiving a word sequence formed by a sequence of words;

forming a plurality of window vectors, each representing a set of z consecutive words in the word sequence;

transforming the window vectors into a plurality of local contextual feature (LCF) vectors, based on a first part of a convolutional latent semantic model;

generating a global feature vector by selecting, for each dimension of the LCF vectors, a maximum value specified by the LCF vectors, with respect to that dimension; and

projecting the global feature vector into a concept vector, based on a second part of the convolutional latent semantic model,

the convolutional latent semantic model being trained based on click-through data.

2. The method of claim 1,

wherein the word sequence is associated with a first linguistic item, and wherein the concept vector comprises a first concept vector,

and wherein the method further comprises:

performing said forming, transforming, generating, and projecting with respect to at least a second linguistic item, to produce a second concept vector; and

comparing the first concept vector with the second concept vector to generate a similarity measure.

3. The method of claim 2, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to a document.

4. The method of claim 2, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to a keyword associated with an advertising system.

5. The method of claim 2, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to an ad.

6. The method of claim 2,

wherein the first linguistic item corresponds to one part of an ad and the second linguistic item correspond to another part of the same ad, and

wherein the similarity measure constitutes a quality measure that provides an indication of quality of the ad.

7. The method of claim 1,

further comprising ranking and/or filtering the first linguistic item and/or the second linguistic item based on a collection of features,

wherein the collection of features includes at least the similarity measure.

8. The method of claim 1, further comprising producing the plurality of window vectors by:

forming a plurality of word vectors, based on a letter-n-gram construction, that represent the words in the word sequence; and

concatenating groupings of the word vectors, each corresponding to z words, to form the window vectors, corresponding to letter-trigram window vectors.

9. The method of claim 1, wherein the convolutional latent semantic model is produced in a training process by maximizing a conditional likelihood of clicked documents, or keywords associated with clicked ads, given specified queries.

10. A computer readable storage medium for storing computer readable instructions, the computer readable instructions implementing a method when executed by one or more processing devices, the method comprising:

receiving a query;

transforming the query into a query concept vector in a high-level semantic space, using a deep learning model;

comparing the query concept vector with an ad-related concept vector, the ad-related concept vector also being expressed in the semantic space, and being produced by transforming an ad-related linguistic item using the deep learning model,

said comparing producing a query-to-item similarity measure indicating a semantic similarity between the query and the ad-related linguistic item.

11. The computer readable storage medium of claim 10, wherein the deep learning model corresponds to a convolutional latent semantic model.

12. The computer readable storage medium of claim 10, wherein the deep learning model is implemented by a convolutional neural network.

13. The computer readable storage medium of claim 10, wherein the ad-related item corresponds to an ad-related keyword.

14. The computer readable storage medium of claim 10, wherein the ad-related item corresponds to an ad.

15. The computer readable storage medium of claim 10, wherein the query corresponds to a word sequence, and wherein said transforming comprises:

forming a plurality of letter-trigram window vectors, each representing a set of z consecutive words in the word sequence;

transforming the letter-trigram window vectors into a plurality of local contextual feature (LCF) vectors, based on a first part of the deep learning model;

generating a global feature vector by selecting, for each dimension of the LCF vectors, a maximum value specified by the LCF vectors, with respect to that dimension; and

projecting the global feature vector into the query concept vector, based on a second part of the deep learning model.

16. The computer readable storage medium of claim 10, wherein the deep learning model is trained based on click-through data.

17. The computer readable storage medium of claim 10,

further comprising generating a ranking score for the ad-related linguistic item using a ranking model, based on a collection of ranking features,

wherein the collection of ranking features includes at least the query-to-item similarity measure and/or the ad-related concept vector.

18. The computer readable storage medium of claim 10,

further comprising filtering the ad-related linguistic item based on a collection of filtering features, to determine whether to exclude the ad-related linguistic item from further consideration,

wherein the collection of filtering features includes at least the query-to-item similarity measure and/or the ad-related concept vector.

19. The computer readable storage medium of claim 10, further comprising:

generating two or more item-part concept vectors, using the deep learning model, based on different parts of the ad-related item;

forming a quality measure based on said two or more item-part concept vectors; and

ranking or filtering the ad-related item based on a collection of features,

wherein the collection of features includes at least the query-to-item similarity measure and/or the quality measure.

20. A system, implemented by one or more computing devices, for processing linguistic items, comprising:

a similarity determination system configured to;

transform a first linguistic item into a first concept vector expressed in a high-level semantic space, using a convolutional neural network, or receive a first concept vector that has been previously produced;

transform a second linguistic item into a second concept vector expressed in the high-level semantic space, using the convolutional neural network, or receive a second concept vector that has been previously produced; and

compare the first concept vector with the second concept vector to produce a similarity measure; and

a post-processing module configured to perform an action on the second linguistic item based on the similarity measure,

the convolutional neural network being trained based on click-through data.