Document classification with weighted supervised n-gram embedding

• US 8,892,488 B2
• Filed: 05/30/2012
• Issued: 11/18/2014
• Est. Priority Date: 06/01/2011
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Qi, Yanjun, Bai, Bing
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Seck, Ababacar
• Application Number:
  US13/483,868
• Publication Number:
  US 20120310627A1
• Time in Patent Office: 902 Days
• Field of Search: None
• US Class Current: 706/45
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Methods and systems for document classification include embedding n-grams from an input text in a latent space, embedding the input text in the latent space based on the embedded n-grams and weighting said n-grams according to spatial evidence of the respective n-grams in the input text, classifying the document along one or more axes, and adjusting weights used to weight the n-grams based on the output of the classifying step.

Claims

1. A method for document classification, comprising:

embedding n-grams from an input text in a latent space;

embedding the input text in the latent space based on the embedded n-grams and weighting the n-grams according to a non-linear function

2. The method of claim 1, wherein embedding the input text in the latent space is calculated as a weighted sum over the embedded n-grams in the input text.

3. The method of claim 1, wherein a weight of each n-gram is modeled as a function of a relative position of each n-gram in the input text and an embedding representation of each n-gram.

4. The method of claim 1, wherein embedding the input text in the latent space further comprises:

dividing the input text into sub-parts;

forming an embedded representation of each of the sub-parts based on embedded n-grams in each respective sub-part; and

concatenating the sub-parts to form an embedded representation of the full input text.

5. The method of claim 4, wherein forming the embedded representation of each subpart includes calculating a weighted sum over the n-grams in the subpart.

6. The method of claim 1, wherein embedding the input text in the latent space comprises calculating a weighted sum over the n-grams in the input text.

7. The method of claim 1, wherein weights for each n-gram are learned by optimizing over a set of training documents with known class labels.

8. The method of claim 7, wherein the weights are learned using a stochastic gradient descent.

9. The method of claim 1, wherein classifying includes applying a classification having three or more classes.

10. A system for document classification, comprising:

an n-gram embedding module configured to embed n-grams from an input text in a latent space;

a document embedding module configured to embed the input the input text in the latent space based on the embedded n-grams, weighted according to a non-linear function

11. The system of claim 10, wherein the document embedding module is further configured to embed the input text in the latent space as a weighted sum over the embedded n-grams in the input text.

12. The system of claim 10, wherein the weight of an n-gram is modeled as a function of both a relative position of the n-gram in the document and an embedding representation of the n-gram.

13. The system of claim 10, wherein the document embedding module is further configured to:

divide the input text into sub-parts;

form an embedded representation of each of the sub-parts based on embedded n-grams in each respective sub-part; and

concatenate the sub-parts to form an embedded representation of the full input text.

14. The system of claim 13, wherein the document embedding module is further configured to form the embedded representation of each subpart by calculating a weighted sum over the n-grams in the subpart.

15. The system of claim 10, wherein the document embedding module is further configured to form the embedded representation of each subpart by calculating a weighted sum over the n-grams in the subpart.

16. The system of claim 10, wherein the weight learning module is further configured to learn weights for each n-gram by optimizing over a set of training documents with known class labels.

17. The system of claim 16, wherein the weight learning module is configured to learn the weights using a stochastic gradient descent.

18. The system of claim 10, wherein the classifier is configured to apply a classification having three or more classes.