Hierarchical deep convolutional neural network for image classification

• US 10,387,773 B2
• Filed: 12/23/2014
• Issued: 08/20/2019
• Est. Priority Date: 10/27/2014
• Status: Active Grant

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: Yan, Zhicheng, Piramuthu, Robinson, Jagadeesh, Vignesh, Di, Wei, Decoste, Dennis
• Primary Examiner(s): Zhen, Li B.
• Assistant Examiner(s): Pellett, Daniel T
• Application Number:
  US14/582,059
• Publication Number:
  US 20160117587A1
• Time in Patent Office: 1,701 Days
• Field of Search: None
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Hierarchical branching deep convolutional neural networks (HD-CNNs) improve existing convolutional neural network (CNN) technology. In a HD-CNN, classes that can be easily distinguished are classified in a higher layer coarse category CNN, while the most difficult classifications are done on lower layer fine category CNNs. Multinomial logistic loss and a novel temporal sparsity penalty may be used in HD-CNN training. The use of multinomial logistic loss and a temporal sparsity penalty causes each branching component to deal with distinct subsets of categories.

Claims

1. A system comprising:

a memory that stores instructions; and

one or more processors configured by the instructions to perform operations comprising;

accessing a dataset comprising categorized data having a plurality of fine categories;

dividing the dataset into a training set and a value set;

training a first convolutional neural network (CNN) model using the training set;

generating a confusion matrix of the first CNN model using the value set;

applying a clustering algorithm to the confusion matrix to determine, for each fine category, an associated coarse category of a plurality of coarse categories, each coarse category being associated with more than one fine category;

training a base CNN to distinguish between the coarse categories;

training a single fine CNN for each coarse category, the single fine CNN for said coarse category to distinguish between the fine categories associated with said coarse category;

receiving a request to classify data;

using the base CNN, determining a coarse category of the data;

using the single fine CNN for the determined coarse category, determining a single fine category of the data; and

responsive to the request, transmitting the single fine category of the data.

2. The system of claim 1, wherein the clustering algorithm comprises an affinity propagation algorithm.

3. The system of claim 2, wherein the operations further comprise:

obtaining low-dimensional feature representations for the fine categories using a Laplacian eigenmap.

4. The system of claim 2, wherein the training of the fine CNN for each coarse category comprises:

training a second CNN model using the training set;

generating the fine CNN for each coarse category from the second CNN; and

training the fine CNN for each coarse category using a subset of the training set, the subset excluding data having fine categories not associated with said coarse category.

5. The system of claim 1, wherein the operations further comprise:

combining the base CNN with each of the fine CNNs to form a hierarchically deep CNN (HD-CNN); and

fine-tuning the HD-CNN.

6. The system of claim 5, wherein the fine-tuning of the HD-CNN comprises:

beginning the fine-tuning with a learning factor;

training the HD-CNN by iterating over a series of training batches using the learning factor;

after each iteration, comparing a training error for the training batch to a threshold;

determining, based on the comparison, that the training error for the training batch is below the threshold; and

in response to the determination that the training error for the training batch is below the threshold, modifying the learning factor.

7. The system of claim 5, wherein the fine-tuning of the HD-CNN comprises:

applying a temporal sparsity penalty term in evaluating each of the fine CNNs.

8. The system of claim 1, wherein the dataset comprising the categorized data comprises categorized images.

16. The system of claim 1, wherein:

the using of the base CNN to determine the coarse category of the data comprises;

providing an input to the base CNN; and

receiving a first output from the base CNN; and

the using of the fine CNN for the determined coarse category to determine the single fine category of the data comprises;

providing the input to the fine CNN; and

receiving a second output from the fine CNN.

17. The system of claim 1, wherein:

the receiving of the request to classify data comprises receiving, from a computing device via a network, an image; and

the operations further comprise;

generating a listing template based on the single fine category; and

causing the listing template to be presented on the computing device.

9. A method comprising:

accessing, by one or more processors, a dataset comprising categorized data having a plurality of fine categories;

dividing, by the one or more processors, the dataset into a training set and a value set;

training, by the one or more processors, a first CNN model using the training set;

generating, by the one or more processors, a confusion matrix of the first CNN model using the value set;

applying, by the one or more processors, a clustering algorithm to the confusion matrix to determine, for each fine category, an associated coarse category of a plurality of coarse categories, each coarse category being associated with more than one fine category;

training, by the one or more processors, a base convolutional neural network (CNN) to distinguish between the coarse categories, the base CNN implemented by a processor of a machine;

training, by the one or more processors, a single fine CNN for each coarse category, the single fine CNN for said coarse category to distinguish between the fine categories associated with said coarse category;

receiving, by the one or more processors, a request to classify data;

using the base CNN, determining, by the one or more processors, a coarse category of the data;

using the single fine CNN for the determined coarse category, determining, by the one or more processors, a single fine category of the data; and

responsive to the request, transmitting, by the one or more processors, the single fine category of the data.

10. The method of claim 9, wherein the clustering algorithm comprises an affinity propagation algorithm.

11. The method of claim 10, further comprising obtaining low-dimensional feature representations for the fine categories using a Laplacian eigenmap.

12. The method of claim 10, wherein the training of the fine CNN for each coarse category comprises:

training a second CNN model using the training set;

generating the fine CNN for each coarse category from the second CNN; and

training the fine CNN for each coarse category using a subset of the training set, the subset excluding data having fine categories not associated with said coarse category.

13. The method of claim 9, further comprising:

combining the base CNN with each of the fine CNNs to form a hierarchically deep CNN (HD-CNN); and

fine-tuning the HD-CNN.

14. The method of claim 13, wherein the fine-tuning of the HD-CNN includes:

beginning the fine-tuning with a learning factor;

training the HD-CNN by iterating over a series of training batches using the learning factor;

after each iteration, comparing a training error for the training batch to a threshold;

determining, based on the comparison, that the training error for the training batch is below the threshold; and

in response to the determination that the training error for the training batch is below the threshold, modifying the learning factor.

15. The method of claim 13, wherein the fine-tuning of the HD-CNN includes:

applying a temporal sparsity penalty term in evaluating each of the fine CNNs.

18. A non-transitory machine-readable medium having instructions embodied thereon, the instructions executable by a processor of a machine to perform operations comprising:

accessing a dataset comprising categorized data having a plurality of fine categories;

dividing the dataset into a training set and a value set;

training a first CNN model using the training set;

generating a confusion matrix of the first CNN model using the value set;

applying a clustering algorithm to the confusion matrix to determine, for each fine category, an associated coarse category of a plurality of coarse categories, each coarse category being associated with more than one fine category;

training a base convolutional neural network (CNN) to distinguish between the coarse categories, the CNN implemented by a processor of a machine;

training a single fine CNN for each coarse category, the single fine CNN for said coarse category to distinguish between the fine categories associated with said coarse category;

receiving a request to classify data;

using the base CNN, determining a coarse category of the data;

using the single fine CNN for the determined coarse category, determining a single fine category of the data; and

responsive to the request, transmitting the single fine category of the data.

19. The non-transitory machine-readable medium of claim 18, wherein the clustering algorithm comprises an affinity propagation algorithm.

20. The non-transitory machine-readable medium of claim 19, wherein the training of the fine CNN for each coarse category comprises:

training a second CNN model using the training set;

generating the fine CNN for each coarse category from the second CNN model; and

training the fine CNN for each coarse category using a subset of the training set, the subset excluding data having fine categories not associated with said coarse category.