Activation layers for deep learning networks

• US 9,892,344 B1
• Filed: 11/30/2015
• Issued: 02/13/2018
• Est. Priority Date: 11/30/2015
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Tran, Son Dinh, Manmatha, Raghavan
• Primary Examiner(s): Krasnic, Bernard
• Application Number:
  US14/954,646
• Time in Patent Office: 806 Days
• Field of Search: None
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

Tasks such as object classification from image data can take advantage of a deep learning process using convolutional neural networks. These networks can include a convolutional layer followed by an activation layer, or activation unit, among other potential layers. Improved accuracy can be obtained by using a generalized linear unit (GLU) as an activation unit in such a network, where a GLU is linear for both positive and negative inputs, and is defined by a positive slope, a negative slope, and a bias. These parameters can be learned for each channel or a block of channels, and stacking those types of activation units can further improve accuracy.

Claims

1. A computer-implemented method, comprising:

obtaining a convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit, the generalized linear unit having a functional form described using a pair of straight lines with three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;

training the convolutional neural network using a first subset of a training data set, the training data set including instances of image data with defined labels;

measuring accuracy of the convolutional neural network after the training by classifying images in a test set;

receiving a query image from a client device associated with a user;

processing the query image using the convolutional neural network to determine a classification of an object represented in the query image;

determining a set of items corresponding to the classification; and

providing, to the client device, information for at least a subset of the set of items.

2. The computer-implemented method of claim 1, wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.

3. The computer-implemented method of claim 1, wherein the three parameters are learnable over a block of input channels.

4. The computer-implemented method of claim 3, wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over the block of input channels.

5. A computer-implemented method, comprising:

training a convolutional neural network using a set of a training data, the training data including instances of image data with determined classifications;

receiving query image data;

processing the query image data using the convolutional neural network to determine a classification of an object represented in the query image data, the convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit (GLU), the GLU having three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;

determining a set of items corresponding to the classification; and

providing, to a client device, information for at least a subset of the set of items.

6. The computer-implemented method of claim 5, further comprising:

measuring accuracy of the convolutional neural network after the training by classifying images in a test set.

7. The computer-implemented method of claim 6, further comprising:

performing a validation of the convolutional neural network after training using a set of validation data.

8. The computer-implemented method of claim 5, wherein the GLU is linear for positive inputs and negative inputs.

9. The computer-implemented method of claim 5, wherein the GLU further includes at least one of a pooling layer, a fully connected layer, or a softmax layer.

10. The computer-implemented method of claim 5, wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.

11. The computer-implemented method of claim 5, wherein the three parameters are learnable over a block of input channels.

12. The computer-implemented method of claim 11, wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over the block of input channels.

13. A system, comprising:

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the system to;

train a convolutional neural network using a set of a training data, the training data including instances of image data with determined classifications;

receive query data;

process the query data using the convolutional neural network to determine a classification of at least a portion of the query data, the convolutional neural network containing at least one convolutional layer and at least one activation layer, the at least one activation layer including a generalized linear unit (GLU), the GLU having three parameters including a first slope in a positive region, a second slope in a negative region, and an offset applied to the first slope and the second slope, the three parameters learnable over at least one input channel;

determine a set of items corresponding to the classification; and

provide, in response to a request, information for at least a subset of the set of items.

14. The system of claim 13, wherein the query data is classified using at least one of a computer vision process, an object recognition process, an audio signal analysis process, a speech recognition process, or an item classification process utilizing the convolutional neural network.

15. The system of claim 13, wherein the instructions when executed further cause the system to:

measure accuracy of the convolutional neural network after the training by classifying images in a test set.

16. The system of claim 13, wherein the GLU further includes at least one of a pooling layer, a fully connected layer, or a softmax layer.

17. The system of claim 13, wherein the first slope is associated with positive inputs and the second slope is associated with negative inputs.

18. The system of claim 13, wherein the three parameters are learnable over a block of input channels.

19. The system of claim 18, wherein a first activation layer with the three parameters learnable over a single channel is stacked with a second activation layer with the three parameters learnable over the block of input channels.

20. The system of claim 13, wherein the GLU is linear for both positive and negative inputs.