MULTI-LAYER FUSION IN A CONVOLUTIONAL NEURAL NETWORK FOR IMAGE CLASSIFICATION

• US 20170140253A1
• Filed: 06/10/2016
• Published: 05/18/2017
• Est. Priority Date: 11/12/2015
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Xu, Beilei, Wshah, Safwan, Bulan, Orhan, Kumar, Jayant, Paul, Peter
• Granted Patent: US 10,068,171 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles

Abstract

A method and system for domain adaptation based on multi-layer fusion in a convolutional neural network architecture for feature extraction and a two-step training and fine-tuning scheme. The architecture concatenates features extracted at different depths of the network to form a fully connected layer before the classification step. First, the network is trained with a large set of images from a source domain as a feature extractor. Second, for each new domain (including the source domain), the classification step is fine-tuned with images collected from the corresponding site. The features from different depths are concatenated with and fine-tuned with weights adjusted for a specific task. The architecture is used for classifying high occupancy vehicle images.

Claims

1. A method of constructing a convolutional neural network (CNN) for domain adaptation utilizing features extracted from multiple levels, including:

selecting a CNN architecture including a plurality of convolutional layers and fully connected layers;

training the CNN on a source domain data set;

selecting a plurality of layers from the plurality of convolutional layers across the trained CNN;

extracting features from the selected layers from the trained CNN;

concatenating the extracted features to form a feature vector;

connecting the feature vector to a fully connected neural network classifier; and

, fine-tuning the fully connected neural network classifier from a target domain data set.

2. The method of claim 1 wherein the selected CNN network can be any CNN network selected from the group consisting of GoogLeNet, AlexNet, VGG-M, VGG-D, and VGG-F.

3. The method of claim 1 wherein the selecting layers includes selecting one or a plurality of layers from a low, medium, or high level portion of the CNN.

4. The method of claim 1 wherein the concatenating features includes a feature vector that can be weighted before the concatenating using a learning rate.

5. The method of claim 1 wherein the connecting the concatenated feature vector includes any type of fully connected neural network including softmax activation.

6. The method of claim 1 wherein the selecting features includes selecting a feature which can be directly extracted from the convolutional and fully connected layer or through additional convolutional and fully connected layers.

7. The method of claim 1 wherein the target domain can be different or the same as the source domain.

8. The method of claim 1 wherein the training and fine-tuning use different datasets.

9. The method of claim 6 wherein the additional convolutional and fully connected layer is trained from a source domain data set with concatenated components and a fully connected neural network classifier.

11. The system of claim 7 wherein the computer is further programmed to:

(i) train the CNN by optimizing weights of the CNN with respect to the source domain training set; and

(ii) after the operation (i), train the combination of the CNN and the features fusion network by optimizing weights of the features fusion network with respect to the target domain training set.

12. The system of claim 8 wherein the features fusion network includes:

a features extraction layer operating to extract features from the network layers of the CNN; and

a concatenation layer that concatenates the extracted features to generate a concatenated features vector representation of the input image;

wherein the weights of the features fusion network include weights of the extracted features in the concatenated features vector.

13. The system of claim 8 wherein the operation (ii) further includes (ii)(a) optimizing weights of the CNN with respect to the target domain training set.

15. The system of claim 7 wherein the features fusion network includes:

a features extraction layer operating to extract features from the network layers of the CNN; and

a concatenation layer that concatenates the extracted features to generate a concatenated features vector representation of the input image.

16. The system of claim 12 wherein the concatenation layer further includes a nonlinear activation function.

17. The system of claim 13 wherein the nonlinear activation function comprises a softmax function.

18. The system of claim 12 wherein the features extraction layer comprises a sequence of layers including, in order:

an average pooling layer;

a convolution layer; and

one or more fully connected layers.

19. The system of claim 7 wherein the computer is further programmed to estimate a vehicle occupancy based on the classification of the input image.

20. The system of claim 7 wherein the source domain training set comprises images of side views of vehicles, the target domain training set comprises images of side views of vehicles, and the computer is further programmed to estimate a vehicle occupancy based on the classification of the input image.

22. The adaptation method of claim 18 further comprising:

training at least some weights of the CNN using the images in a target domain different from the source domain concurrently with the training of the weights of the weighted combination of inputted features.

23. The adaptation method of claim 19 wherein the training of at least some weights of the CNN preferentially adjusts weights of higher network levels of the CNN.

10. An image classification system comprising:

a computer programmed to perform classification of an input image from a target domain by operations including;

processing the input image using a convolutional neural network (CNN) having a plurality of network layers and trained on a source domain training set;

processing outputs of at least a fraction of the plurality of network layers of the CNN using a features fusion network trained on a target domain training set to generate a classification of the input image.

14. The system of claim 10 wherein the operation (ii)(a) more strongly optimizes weights of higher network layers of the CNN compared with lower network layers of the CNN.

21. A method of adapting a convolutional neural network (CNN) trained to classify images of a source domain to a target domain, the adaptation method comprising:

inputting features output by at least a fraction of the network levels of the CNN into a features fusion network outputting a weighted combination of the inputted features; and

training weights of the weighted combination of inputted features using images in a target domain different from the source domain.

24. A method of operating a convolutional neural network (CNN) for image classification across multiple domains, comprising:

capturing a data representation of an image via an imaging device;

processing the data representation in the CNN to extract features therein relevant to the image classification including associating CNN levels with the extracted features, and fusing selected ones of the levels in a concatenated features network, wherein the selected ones of the levels have a selective weighting for effecting the image classification; and

,

determining an output image classification for the image from the concatenated features network wherein the fusion of the selected ones of the levels varies among the multiple domains.