SEMISUPERVISED AUTOENCODER FOR SENTIMENT ANALYSIS

• US 20180165554A1
• Filed: 12/11/2017
• Published: 06/14/2018
• Est. Priority Date: 12/09/2016
• Status: Active Grant

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Zhang, Zhongfei, Zhai, Shuangfei
• Granted Patent: US 11,205,103 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting

Abstract

A method of modelling data, comprising: training an objective function of a linear classifier, based on a set of labeled data, to derive a set of classifier weights; defining a posterior probability distribution on the set of classifier weights of the linear classifier; approximating a marginalized loss function for an autoencoder as a Bregman divergence, based on the posterior probability distribution on the set of classifier weights learned from the linear classifier; and classifying unlabeled data using the autoencoder according to the marginalized loss function.

Claims

1. A method of modelling data, comprising:

training an objective function of a linear classifier, based on a set of labeled data, to derive a set of classifier weights;

defining a posterior probability distribution on the set of classifier weights of the linear classifier;

approximating a marginalized loss function for an autoencoder as a Bregman divergence, based on the posterior probability distribution on the set of classifier weights learned from the linear classifier; and

automatically classifying unlabeled data using a compact classifier according to the marginalized loss function.

2. The method according to claim 1, wherein the data comprises data selected from the group consisting of semantic data, text data, and text documents.

3. The method according to claim 1, wherein the autoencoder comprises a neural network, wherein said training comprises training the neural network.

4. The method according to claim 1, wherein the autoencoder comprises a denoising autoencoder.

5. The method according to claim 4, wherein the denoising autoencoder is denoised stochastically, and comprises a neural network trained according to stochastic gradient descent training using randomly selected data samples, wherein a gradient is calculated using back propagation of errors.

6. The method according to claim 1, wherein said training comprises training the objective function of the linear classifier with a bag of words, wherein the linear classifier comprises a support vector machine classifier with squared hinge loss and l₂ regularization.

7. The method according to claim 1, wherein said training comprises training the objective function of the linear classifier with a bag of words, wherein the linear classifier comprises a Logistic Regression classifier.

8. The method according to claim 1, wherein the Bregman divergence is determined assuming that all data samples induce a loss.

9. The method according to claim 1, wherein the posterior probability distribution on the set of classifier weights is estimated using with a Laplace approximation, wherein the Laplace approximation stochastically estimates the set of classifier weights using a covariance matrix constrained to be diagonal.

10. The method according to claim 1, wherein the posterior probability distribution on the set of classifier weights is estimated using with a Markov chain Monte Carlo method.

11. A system for modelling data, comprising:

an input port, configured to receive a set of labelled data;

a linear classifier;

an autoencoder;

a compact classifier; and

an output port, configured to communicate a classification of at least one unlabeled datum,

wherein;

an objective function of a linear classifier is automatically trained, based on the set of labeled data, to derive a set of classifier weights;

a marginalized loss function for the autoencoder approximated as a Bregman divergence, based on a posterior probability distribution on the set of classifier weights learned from the linear classifier; and

the at least one unlabeled datum classified using the compact classifier according to the marginalized loss function.

12. The system according to claim 11, wherein the data is selected from the group consisting of semantic data, text data, and text documents.

13. The system according to claim 11, wherein the autoencoder comprises a neural network.

14. The system according to claim 11, wherein the autoencoder comprises a denoising autoencoder.

15. The system according to claim 14, wherein the denoising autoencoder is denoised stochastically, and comprises a neural network trained according to stochastic gradient descent training using randomly selected data samples, wherein a gradient is calculated using back propagation of errors.

16. The system according to claim 11, wherein the objective function of the linear classifier is trained with a bag of words, wherein the linear classifier comprises a support vector machine classifier with squared hinge loss and l₂ regularization.

17. The system according to claim 11, wherein the objective function of the linear classifier is trained with a bag of words, wherein the linear classifier comprises a Logistic Regression classifier.

18. The system according to claim 11, wherein the Bregman divergence is determined assuming that all data samples induce a loss.

19. The system according to claim 11, wherein the posterior probability distribution on the set of classifier weights is automatically estimated using a technique selected from the group consisting of a Laplace approximation, wherein the Laplace approximation stochastically estimates the set of classifier weights using a covariance matrix constrained to be diagonal, and a Markov chain Monte Carlo method.

20. A computer readable medium containing non-transitory instructions for controlling at least one programmable automated processor to perform a method of modelling data, comprising:

training an objective function of a linear classifier, based on a set of labeled data, to derive a set of classifier weights;

defining a posterior probability distribution on the set of classifier weights of the linear classifier;

approximating a marginalized loss function for an autoencoder as a Bregman divergence, based on the posterior probability distribution on the set of classifier weights learned from the linear classifier; and

classifying unlabeled data using a compact classifier according to the marginalized loss function.