TEXT CLASSIFICATION BY RANKING WITH CONVOLUTIONAL NEURAL NETWORKS

• US 20170308790A1
• Filed: 04/21/2016
• Published: 10/26/2017
• Est. Priority Date: 04/21/2016
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Nogueira dos Santos, Cicero, Xiang, Bing, Zhou, Bowen
• Application Number:
  US15/134,719
• Publication Number:
  US 20170308790A1
• Time in Patent Office: Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

According to an aspect a method includes configuring a convolutional neural network (CNN) for classifying text based on word embedding features into a predefined set of classes identified by class labels. The predefined set of classes includes a class labeled none-of-the-above for text that does not fit into any of the other classes in the predefined set of classes. The CNN is trained based on a set of training data. The training includes learning parameters of class distributed vector representations (DVRs) of each of the predefined set of classes. The learning includes minimizing a pair-wise ranking loss function over the set of training data. A class embedding matrix of the class DVRs of the predefined set of classes that excludes a class embedding for the none-of-the-above class is generated. Each column in the class embedding matrix corresponds to one of the predefined classes.

Claims

1. A method comprising:

configuring a convolutional neural network (CNN) for classifying text based on word embedding features into a predefined set of classes identified by class labels, the predefined set of classes including a class that is labeled none-of-the-above for text that does not fit into any of the other classes in the predefined set of classes, the configuring comprising;

receiving a set of training data that includes for each training round;

training text, a correct class label that correctly classifies the training text, and an incorrect class label that incorrectly classifies the training text, the correct class label and the incorrect class label selected from the class labels that identify the predefined set of classes;

training the CNN based on the set of training data, the training including;

learning parameters of class distributed vector representations (DVRs) of each of the predefined set of classes, the learning including minimizing a pair-wise ranking loss function over the set of training data and causing the CNN to generate;

a score of less than zero in response to a correct class label of none-of-the-above, and a score of greater than zero in response to a correct class label having any other value; and

a score of less than zero in response to an incorrect class label; and

generating a class embedding matrix of the class DVRs of the predefined set of classes that excludes a class embedding for the none-of-the-above class, each column in the class embedding matrix corresponding to one of the predefined classes.

2. The method of claim 1, wherein the score that is greater than zero is greater than zero by a first specified margin magnified by a scaling margin and the score that is less than zero is less than zero by a second specified margin magnified by the scaling margin.

3. The method of claim 1, wherein stochastic gradient descent with back propagation is used to update the parameters.

4. The method of claim 1, wherein input features to the CNN include word embeddings of one or more words in each set of training text.

5. The method of claim 1, wherein the set of classes include relations between nouns in the input text.

6. The method of claim 1, wherein the set of classes include sentiments of the input text.

7. The method of claim 1, further comprising:

receiving, by the CNN, a text string;

predicting, by the CNN, a class label of the text string.

8. The method of claim 7, wherein the predicting comprises:

generating a DVR of the text string;

comparing the DVR of the text string to the class DVRs in the class embedding matrix to generate a score for each of the classes corresponding to columns in the class embedding matrix;

selecting the highest generated score;

based on the selected score being a positive number, outputting the class label corresponding to the selected score as the predicted class label of the text string; and

based on the selected score being a negative number, outputting the class label of none-of-the-above as the predicated class label of the text string.

9. A system comprising:

a memory having computer readable computer instructions; and

a processor for executing the computer readable instructions, the computer readable instructions including;

configuring a convolutional neural network (CNN) for classifying text based on word embedding features into a predefined set of classes identified by class labels, the predefined set of classes including a class that is labeled none-of-the-above for text that does not fit into any of the other classes in the predefined set of classes, the configuring comprising;

receiving a set of training data that includes for each training round;

training text, a correct class label that correctly classifies the training text, and an incorrect class label that incorrectly classifies the training text, the correct class label and the incorrect class label selected from the class labels that identify the predefined set of classes;

training the CNN based on the set of training data, the training including;

learning parameters of class distributed vector representations (DVRs) of each of the predefined set of classes, the learning including minimizing a pair-wise ranking loss function over the set of training data and causing the CNN to generate;

a score of less than zero in response to a correct class label of none-of-the-above, and a score of greater than zero in response to a correct class label having any other value; and

a score of less than zero in response to an incorrect class label; and

generating a class embedding matrix of the class DVRs of the predefined set of classes that excludes a class embedding for the none-of-the-above class, each column in the class embedding matrix corresponding to one of the predefined classes.

10. The system of claim 9, wherein the score that is greater than zero is greater than zero by a first specified margin magnified by a scaling margin and the score that is less than zero is less than zero by a second specified margin magnified by the scaling margin.

11. The system of claim 9, wherein stochastic gradient descent with back propagation is used to update the parameters.

12. The system of claim 9, wherein input features to the CNN include word embeddings of one or more words in each set of training text.

13. The system of claim 9, wherein the instructions further include:

receiving, by the CNN, a text string;

predicting, by the CNN, a class label of the text string.

14. The system of claim 13, wherein the predicting comprises:

generating a DVR of the text string;

comparing the DVR of the text string to the class DVRs in the class embedding matrix to generate a score for each of the classes corresponding to columns in the class embedding matrix;

selecting the highest generated score;

based on the selected score being a positive number, outputting the class label corresponding to the selected score as the predicted class label of the text string; and

based on the selected score being a negative number, outputting the class label of none-of-the-above as the predicated class label of the text string.

15. A computer program product comprising:

a tangible storage medium readable by a processor and storing instructions executable by the processor for;

configuring a convolutional neural network (CNN) for classifying text based on word embedding features into a predefined set of classes identified by class labels, the predefined set of classes including a class that is labeled none-of-the-above for text that does not fit into any of the other classes in the predefined set of classes, the configuring comprising;

receiving a set of training data that includes for each training round;

training text, a correct class label that correctly classifies the training text, and an incorrect class label that incorrectly classifies the training text, the correct class label and the incorrect class label selected from the class labels that identify the predefined set of classes;

training the CNN based on the set of training data, the training including;

learning parameters of class distributed vector representations (DVRs) of each of the predefined set of classes, the learning including minimizing a pair-wise ranking loss function over the set of training data and causing the CNN to generate;

a score of less than zero in response to a correct class label of none-of-the-above, and a score of greater than zero in response to a correct class label having any other value; and

a score of less than zero in response to an incorrect class label; and

generating a class embedding matrix of the class DVRs of the predefined set of classes that excludes a class embedding for the none-of-the-above class, each column in the class embedding matrix corresponding to one of the predefined classes.

16. The computer program product of claim 15, wherein the score that is greater than zero is greater than zero by a first specified margin magnified by a scaling margin and the score that is less than zero is less than zero by a second specified margin magnified by the scaling margin.

17. The computer program product of claim 15, wherein stochastic gradient descent with back propagation is used to update the parameters.

18. The computer program product of claim 15, wherein input features to the CNN include word embeddings of one or more words in each set of training text.

19. The computer program product of claim 15, wherein the instructions are further executable by the processor for:

receiving, by the CNN, a text string;

predicting, by the CNN, a class label of the text string.

20. The computer program product of claim 19, wherein the predicting comprises:

generating a DVR of the text string;

comparing the DVR of the text string to the class DVRs in the class embedding matrix to generate a score for each of the classes corresponding to columns in the class embedding matrix;

selecting the highest generated score;

based on the selected score being a positive number, outputting the class label corresponding to the selected score as the predicted class label of the text string; and

based on the selected score being a negative number, outputting the class label of none-of-the-above as the predicated class label of the text string.