Activation layers for deep learning networks

• US 10,366,313 B2
• Filed: 02/12/2018
• Issued: 07/30/2019
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Tran, Son Dinh, Manmatha, Raghavan
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US15/894,867
• Publication Number:
  US 20180197049A1
• Time in Patent Office: 533 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Tasks such as object classification from image data can take advantage of a deep learning process using convolutional neural networks. These networks can include a convolutional layer followed by an activation layer, or activation unit, among other potential layers. Improved accuracy can be obtained by using a generalized linear unit (GLU) as an activation unit in such a network, where a GLU is linear for both positive and negative inputs, and is defined by a positive slope, a negative slope, and a bias. These parameters can be learned for each channel or a block of channels, and stacking those types of activation units can further improve accuracy.

Claims

1. A computer-implemented method, comprising:

training a convolutional neural network using a set of a training data, the training data including instances of data with determined classifications;

receiving a query from a client device associated with a user;

processing the query using the trained convolutional neural network to determine a classification of a data object represented in the query, the trained convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit, the generalized linear unit having a functional form described using a pair of straight lines with three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;

determining a set of features corresponding to the classification; and

providing, to the client device, information for at least a subset of the set of features.

2. The computer-implemented method of claim 1, wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.

3. The computer-implemented method of claim 1, claim wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over a block of input channels.

4. A computer-implemented method, comprising:

training a convolutional neural network using a set of a training data, the training data including instances of data objects with determined classifications;

receiving a query;

processing the query using the trained convolutional neural network to determine a classification of a data segment represented in the query, the trained convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit (GLU), the GLU having three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;

determining a set of data segments corresponding to the classification; and

providing, to a client device, information for at least a subset of the set of data segments.

5. The computer-implemented method of claim 4, further comprising:

measuring accuracy of the trained convolutional neural network by classifying data objects in a test set.

6. The computer-implemented method of claim 4, wherein the GLU is linear for positive inputs and negative inputs.

7. The computer-implemented method of claim 4, wherein the GLU further includes at least one of a pooling layer, a fully connected layer, or a softmax layer.

8. The computer-implemented method of claim 4, wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.

9. The computer-implemented method of claim 4, wherein the query comprises an image, an audio sample, a text input, or a combination thereof.

10. The computer-implemented method of claim 9, wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over a block of input channels.

11. A system, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

train a convolutional neural network using a set of a training data, the training data including instances of data with determined classifications;

receive query data;

process the query data using the trained convolutional neural network to determine a classification of at least a portion of the query data, the trained convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit (GLU), the GLU having three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;

determine a set of data objects corresponding to the classification; and

provide, in response to a request, information for at least a subset of the set of data objects.

12. The system of claim 11, wherein the query data is classified using at least one of a computer vision process, an object recognition process, an audio signal analysis process, a speech recognition process, or an item classification process utilizing the trained convolutional neural network.

13. The system of claim 11, wherein the instructions when executed further cause the system to:

measure accuracy of the trained convolutional neural network after the training by classifying data in a test set.

14. The system of claim 11, wherein the GLU further includes at least one of a pooling layer, a fully connected layer, or a softmax layer.

15. The system of claim 11, wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.

16. The system of claim 11, wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over a block of input channels.

17. The system of claim 11, wherein the GLU is linear for both positive and negative inputs.