Implementing a classification model for recognition processing

• US 10,529,318 B2
• Filed: 07/31/2015
• Issued: 01/07/2020
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kurata, Gakuto
• Primary Examiner(s): Afshar, Kamran
• Assistant Examiner(s): Buss, Benjamin J
• Application Number:
  US14/815,564
• Publication Number:
  US 20170032244A1
• Time in Patent Office: 1,621 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks

Abstract

A method, system, and computer program product for learning a recognition model for recognition processing. The method includes preparing one or more examples for learning, each of which includes an input segment, an additional segment adjacent to the input segment and an assigned label. The input segment and the additional segment are extracted from an original training data. A classification model is trained, using the input segment and the additional segment in the examples, to initialize parameters of the classification model so that extended segments including the input segment and the additional segment are reconstructed from the input segment. Then, the classification model is tuned to predict a target label, using the input segment and the assigned label in the examples, based on the initialized parameters. At least a portion of the obtained classification model is included in the recognition model.

Claims

1. A method for learning a recognition model for recognition processing, the method comprising:

preparing one or more examples for learning, each of which includes an input segment, an additional segment adjacent to the input segment and an assigned label, the input segment and the additional segment being extracted from an original training data;

training a classification model, using a processor, using the input segment and the additional segment in the examples to initialize parameters of the classification model to provide initialized parameters so that extended segments, including the input segment and the additional segment, are reconstructed from the input segment without use of further segments to provide reconstructed extended segments; and

tuning the classification model to predict a target label, using the input segment and the assigned label in the examples, based on the initialized parameters, at least a portion of the classification model being included in the recognition model.

2. The method of claim 1, wherein the recognition model includes all portions of the classification model, the classification model estimating posterior probabilities over targets.

3. The method of claim 1, wherein the recognition model uses the at least portion of the classification model as a feature extraction model for a feature extractor and includes a post-stage recognition model, the feature extractor outputting estimated target probabilities or activations of an internal layer of the classification model as features for the post-stage recognition model, the post-stage recognition model estimating posterior probabilities over targets based on the features.

4. The shod of claim 1, wherein a size of input for recognition to the classification model is equal to a size of the input segment for learning.

5. The method of claim 1, wherein information outside the input segment is subsumed in the classification model by treating the input segment as input and the extended segments as prediction of the input during the training.

6. The method of claim 1, wherein the training includes:

optimizing forward mapping parameters and reverse mapping parameters of a layer in the classification model such that a discrepancy between the extended segments and the reconstructed extended segments from the input segment is minimized, the reverse mapping parameters being discarded in response to stacking the layer within the classification model.

7. The method of claim 6, wherein a regularization term is added to a loss function measuring the discrepancy, the regularization term penalizing larger values of the reverse mapping parameters so as to subsume more information h forward mapping parameters than the reverse mapping parameters.

8. The method of claim 1, wherein the classification model includes a deep neural network having one or more hidden layers between an input layer for the input segment and an output layer for targets, the training is included in an unsupervised pre-training process that stacks the one or more hidden layers, the input layer, and the output layer with initializing parameters, the tuning is a fine-tuning process that discriminatively updates the parameters of the layers, the deep neural network is incorporated into a hidden Markov model (HMM) and the targets of the classification model are HMM states.

9. The method of claim 1, wherein the original training data is acoustic data, the input segment is n-frame acoustic features, the extended segment is n+m-frame acoustic features, and the additional segment is m-frame acoustic features preceding and/or succeeding the n-frame acoustic features.

10. The method of claim 1, wherein the original training data is image data, the input segment is a x*y pixel patch, the extended segment is (x+a)*(y+b) pixel patch, and the additional segment is a (b*x+a*y+a*b) pixel patch surrounding the x*y pixel patch.

11. The method of claim 1, wherein at least one of the training and the tuning of the classification model is performed in a cloud corrupting environment.

12. A computer system for learning a recognition model for recognition processing by executing program instructions tangibly stored in a memory, the computer system comprising:

a processor in communication with the memory, wherein the computer system is configured to;

prepare one or more examples for learning, each of which includes an input segment, an additional segment adjacent to the input segment and an assigned label, the input segment and the additional segment being extracted from an original training data;

train a classification model, using the input segment and the additional segment in the examples to initialize parameters of the classification model to provide initialized parameters so that extended segments, including the input segment and the additional segment, are reconstructed from the input segment without use of further segments to provide reconstructed extended segments; and

tune the classification model to predict a target label, using the input segment and the assigned label in the examples, based on the initialized parameters, at least a portion of the classification model being included in the recognition model.

13. The computer system of claim 12, wherein the recognition model includes a whole of the classification model, the classification model estimating posterior probabilities over targets.

14. The computer system of claim 12, wherein the recognition model uses the at least a portion of the classification model as a feature extraction model for a feature extractor and includes a post-stage recognition model, the feature extractor outputting estimated target probabilities or activations of an internal layer of the classification model as features for the post-stage recognition model, the post-stage recognition model estimating posterior probabilities over targets based on the features.

15. The computer system of claim 12, wherein a size of input for recognition to the classification model is equal to a size of the input segment for learning.

16. The computer system of claim 12, wherein information outside the input segment is subsumed in the classification model by treating the input segment as input and the extended segments as prediction of the input during the training.

17. The computer system of claim 12, wherein the computer system is further configured to:

optimize forward mapping parameters and reverse mapping parameters of a layer in the classification model such that a discrepancy between the extended segments and the reconstructed extended segments from the input segment is minimized, the reverse mapping parameters being discarded in response to stacking the layer within the classification model.

18. The computer system of claim 17, wherein a regularization term is added to a loss function measuring the discrepancy, the regularization term penalizing larger value of the reverse mapping parameters so as to subsume more information into the forward mapping parameters than the reverse mapping parameters.

19. The computer system of claim 12, wherein the computer system is provided in a cloud computing environment.

20. A computer program product for learning a recognition model for recognition processing, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

preparing one or more examples for learning, each of which includes an input segment, an additional segment adjacent to the input segment and an assigned label, the input segment and the additional segment being extracted from an original training data;

training a classification model, using a processor, using the input segment and the additional segment in the one or more examples to initialize parameters of the classification model to provide initialized parameters so that extended segments, including the input segment and the additional segment, are reconstructed from the input segment without use of further segments to provide reconstructed extended segments; and

tuning the classification model to predict a target label, using the input segment and the assigned label in the one or more examples, based on the initialized parameters, at least a portion of the classification model being included in the recognition model.

21. The computer program product of claim 20, wherein the recognition model includes a whole of the classification model, the classification model estimating posterior probabilities over targets.

22. The computer program product of claim 20, wherein the recognition model uses the at least a portion of the classification model as a feature extraction model for a feature extractor and includes a post-stage recognition model, the feature extractor outputting estimated target probabilities or activations of an internal layer of the classification model as features for the post-stage recognition model, the post-stage recognition model estimating posterior probabilities over targets based on the features.

23. A method for learning a feature extraction model for recognition processing, the method comprising:

preparing one or more examples for learning, each of which includes an input segment, an additional segment adjacent to the input segment and an assigned label, the input segment and the additional segment being extracted from an original training data;

training a classification model, using a processor, using the input segment and the additional segment in the examples to initialize parameters of the classification model to provide initialized parameters so that extended segments, including the input segment and the additional segment, are reconstructed from the input segment to provide reconstructed extended segments, wherein the training includes;

optimizing forward mapping parameters and reverse mapping parameters of a layer in the classification model such that a discrepancy between the extended segments and the reconstructed extended segments from the input segment is minimized, the reverse mapping parameters being discarded in response to stacking the layer within the classification model, wherein a regularization term is added to a loss function measuring the discrepancy, the regularization term penalizing larger values of the reverse mapping parameters so as to subsume more information into the forward mapping parameters than the reverse mapping parameters;

tuning the classification model, using the input segment and the assigned label in the examples; and

storing at least a portion of the classification model as the feature extraction model for a feature extractor, the feature extractor outputting, based on input, estimated target probabilities or activations of an internal layer of the classification model as features for a post-stage recognition model.

24. A method for learning a classification model for recognition processing, the method comprising:

preparing one or more examples for learning, each of which includes an input segment, an additional segment adjacent to the input segment and an assigned label, the input segment and the additional segment being extracted from an original training data;

training the classification model, using a processor, using the input segment and the additional segment in the examples to initialize parameters of the classification model to provide initialized parameters so that extended segments, including the input segment and the additional segment, are reconstructed from the input segment without use of further segments to provide reconstructed extended segments;

tuning the classification model to predict a target label, using the input segment and the assigned label in the examples, based on the initialized parameters; and

storing the classification model, the classification model estimating, based on input, posterior probabilities over targets.