Multi-layer fusion in a convolutional neural network for image classification
First Claim
1. A method of training a convolutional neural network (CNN) for domainadaptation utilizing features extracted from multiple levels, including:
- selecting a CNN architecture including a plurality of convolutional layers and fully connected layers;
training the CNN on a source domain data set;
selecting a plurality of layers from the plurality of convolutional layers across the trained CNN;
extracting features from the selected layers from the trained CNN;
concatenating the extracted features to form a feature vector;
connecting the feature vector to a fully connected neural network classifier; and
,fine-tuning the fully connected neural network classifier from a target domain data setby optimizing weights of the CNN with respect to the target domain data set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN.
6 Assignments
0 Petitions
Accused Products
Abstract
A method and system for domain adaptation based on multi-layer fusion in a convolutional neural network architecture for feature extraction and a two-step training and fine-tuning scheme. The architecture concatenates features extracted at different depths of the network to form a fully connected layer before the classification step. First, the network is trained with a large set of images from a source domain as a feature extractor. Second, for each new domain (including the source domain), the classification step is fine-tuned with images collected from the corresponding site. The features from different depths are concatenated with and fine-tuned with weights adjusted for a specific task. The architecture is used for classifying high occupancy vehicle images.
15 Citations
16 Claims
-
1. A method of training a convolutional neural network (CNN) for domain
adaptation utilizing features extracted from multiple levels, including: -
selecting a CNN architecture including a plurality of convolutional layers and fully connected layers; training the CNN on a source domain data set; selecting a plurality of layers from the plurality of convolutional layers across the trained CNN; extracting features from the selected layers from the trained CNN; concatenating the extracted features to form a feature vector; connecting the feature vector to a fully connected neural network classifier; and
,fine-tuning the fully connected neural network classifier from a target domain data set by optimizing weights of the CNN with respect to the target domain data set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN. - View Dependent Claims (2, 3, 4, 5, 6, 11)
-
-
7. An image classification system comprising:
a computer programmed to perform classification of an input image from a target domain by operations including; processing the input image using a convolutional neural network (CNN) having a plurality of network layers and trained on a source domain training set; processing outputs of at least a fraction of the plurality of network layers of the CNN using a features fusion network trained on a target domain training set to generate a classification of the input image; training the CNN by optimizing weights of the CNN with respect to the source domain training set and training the combination of the CNN and the features fusion network by optimizing weights of the features fusion network with respect to the target domain training set wherein the features fusion network includes; a features extraction layer operating to extract features from the network layers of the CNN; a concatenation layer that concatenates the extracted features to generate a concatenated features vector representation of the input image; and wherein the weights of the features fusion network include weights of the extracted features in the concatenated features vector; and
,optimizing weights of the CNN with respect to the target domain training set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN. - View Dependent Claims (12, 14, 15, 16)
-
8. A method of adapting a convolutional neural network (CNN) trained to classify images of a source domain to a target domain, the adaptation method comprising:
-
inputting features output by at least a fraction of the network levels of the CNN into a features fusion network outputting a weighted combination of the inputted features; and training weights of the weighted combination of inputted features using images in a target domain different from the source domain; and classifying the image in accordance with the trained weights by optimizing weights of the CNN with respect to the target domain data set by more strongly optimizing weights of higher network layers of the CNN compared with lower network layers of the CNN. - View Dependent Claims (9, 10, 13)
-
Specification