Convolutional latent semantic models and their applications

• US 9,477,654 B2
• Filed: 04/01/2014
• Issued: 10/25/2016
• Est. Priority Date: 04/01/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Xiaodong, Gao, Jianfeng, Deng, Li, Lou, Qiang, Zhou, Yunhong, Liu, Guowei, Buehrer, Gregory T., Mao, Jianchang, Shen, Yelong, Zhang, Ruofei
• Primary Examiner(s): LU, KUEN S
• Application Number:
  US14/242,845
• Publication Number:
  US 20150278200A1
• Time in Patent Office: 938 Days
• Field of Search: 707/749
• US Class Current: 1/1
• CPC Class Codes: G06F 40/194 Calculation of difference between files
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Functionality is described herein for transforming first and second symbolic linguistic items into respective first and second continuous-valued concept vectors, using a deep learning model, such as a convolutional latent semantic model. The model is designed to capture both the local and global linguistic contexts of the linguistic items. The functionality then compares the first concept vector with the second concept vector to produce a similarity measure. More specifically, the similarity measure expresses the closeness between the first and second linguistic items in a high-level semantic space. In one case, the first linguistic item corresponds to a query, and the second linguistic item may correspond to a phrase, or a document, or a keyword, or an ad, etc. In one implementation, the convolutional latent semantic model is produced in a training phase based on click-through data.

Claims

1. A method, implemented by one or more computing devices, for processing linguistic items, comprising:

receiving a word sequence formed by a sequence of words;

forming a plurality of window vectors, each representing a set of z consecutive words in the word sequence;

transforming the window vectors into a plurality of local contextual feature (LCF) vectors, based on a first part of a convolutional latent semantic model;

generating a global feature vector by selecting, for each dimension of the LCF vectors, a maximum value specified by the LCF vectors, with respect to that dimension; and

projecting the global feature vector into a concept vector, based on a second part of the convolutional latent semantic model, the convolutional latent semantic model being trained based on click-through data.

2. The method of claim 1, wherein the word sequence is associated with a first linguistic item, and wherein the concept vector comprises a first concept vector,

and wherein the method further comprises:

performing said forming, transforming, generating, and projecting with respect to at least a second linguistic item, to produce a second concept vector; and

comparing the first concept vector with the second concept vector to generate a similarity measure.

3. The method of claim 2, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to a document.

4. The method of claim 2, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to a keyword associated with an advertising system.

5. The method of claim 2, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to an ad.

6. The method of claim 2, wherein the first linguistic item corresponds to one part of an ad and the second linguistic item correspond to another part of the same ad, and

wherein the similarity measure constitutes a quality measure that provides an indication of quality of the ad.

7. The method of claim 2,

further comprising ranking and/or filtering the first linguistic item and/or the second linguistic item based on a collection of features,

wherein the collection of features includes at least the similarity measure.

8. The method of claim 1, further comprising producing the plurality of window vectors by:

forming a plurality of word vectors, based on a letter-n-gram construction, that represent the words in the word sequence; and

concatenating groupings of the word vectors, each corresponding to z words, to form the window vectors, corresponding to letter-trigram window vectors.

9. The method of claim 1, wherein the convolutional latent semantic model is produced in a training process by maximizing a conditional likelihood of clicked documents, or keywords associated with clicked ads, given specified queries.

10. A device comprising:

a processor; and

executable instructions operable by the processor, the executable instructions comprising a method for processing linguistic items, the method comprising;

receiving a word sequence formed by a sequence of words;

forming a plurality of window vectors, each representing a set of z consecutive words in the word sequence;

transforming the window vectors into a plurality of local contextual feature (LCF) vectors, based on a first part of a convolutional latent semantic model;

generating a global feature vector by selecting, for each dimension of the LCF vectors, a maximum value specified by the LCF vectors, with respect to that dimension; and

projecting the global feature vector into a concept vector, based on a second part of the convolutional latent semantic model, the convolutional latent semantic model being trained based on click-through data.

11. The device of claim 10, wherein the word sequence is associated with a first linguistic item and the concept vector comprises a first concept vector,

and wherein the method further comprises:

performing said forming, transforming, generating, and projecting with respect to at least a second linguistic item, to produce a second concept vector; and

comparing the first concept vector with the second concept vector to generate a similarity measure.

12. The device of claim 11, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to a document.

13. The device of claim 11, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to an ad.

14. The device of claim 11, wherein the method further comprises ranking the first linguistic item and the second linguistic item based on a collection of features.

15. The device of claim 10, wherein the method further comprises producing the plurality of window vectors by:

forming a plurality of word vectors, based on a letter-n-gram construction, that represent the words in the word sequence; and

concatenating groupings of the word vectors, each corresponding to z words, to form the window vectors, corresponding to letter-trigram window vectors.

16. An apparatus comprising:

a memory storing processor-executable instructions; and

a processor operatively coupled to the memory, wherein upon execution of the processor-executable instructions, the processor;

receives a word sequence formed by a sequence of words;

forms a plurality of window vectors, each representing a set of z consecutive words in the word sequence;

transforms the window vectors into a plurality of local contextual feature (LCF) vectors, based on a first part of a convolutional latent semantic model;

generates a global feature vector by selecting, for each dimension of the LCF vectors, a maximum value specified by the LCF vectors, with respect to that dimension; and

projects the global feature vector into a concept vector, based on a second part of the convolutional latent semantic model, the convolutional latent semantic model being trained based on click-through data.

17. The apparatus of claim 16, wherein the word sequence is associated with a first linguistic item and the concept vector comprises a first concept vector,

and wherein the processor further:

performs said forming, transforming, generating, and projecting with respect to at least a second linguistic item, to produce a second concept vector; and

compares the first concept vector with the second concept vector to generate a similarity measure.

18. The apparatus of claim 17, wherein the first linguistic item corresponds to a query and the second linguistic item corresponds to a keyword associated with an advertising system.

19. The apparatus of claim 17, wherein the first linguistic item corresponds to one part of an ad and the second linguistic item correspond to another part of the same ad, and

wherein the similarity measure constitutes a quality measure that provides an indication of quality of the ad.

20. The apparatus of claim 17, wherein the processor further filters the first linguistic item and the second linguistic item based on a collection of features.