Methods for object localization and image classification

• US 10,318,848 B2
• Filed: 08/25/2016
• Issued: 06/11/2019
• Est. Priority Date: 12/15/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Dijkman, Daniel Hendricus Franciscus, Julian, David Jonathan
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): Rice, Elisa M
• Application Number:
  US15/247,805
• Publication Number:
  US 20170169314A1
• Time in Patent Office: 1,020 Days
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/2163 Partitioning the feature space
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space

Abstract

A method of training for image classification includes labelling a crop from an image including an object of interest. The crop may be labelled with an indication of whether the object of interest is framed, partially framed or not present in the crop. The method may also include assigning a fully framed class to the labelled crop, including the object of interest, if the object of interest is framed. A labelled crop may be assigned a partially framed class if the object of interest is partially framed. A background class may be assigned to a labelled crop if the object of interest is not present in the crop.

Claims

1. A method for image classification, comprising:

labelling, by a first artificial neural network, a crop from an image including an object of interest with an indication of whether the object of interest is framed, partially framed, or not present in the crop;

assigning a fully framed class to the labelled crop, including the object of interest, when the object of interest is framed;

assigning a partially framed class to the labelled crop, including the object of interest, when the object of interest is partially framed;

assigning a background class to the labelled crop when the object of interest is not present in the crop; and

classifying the object of interest in the image based on a class assigned to the labelled crop.

2. The method of claim 1, further comprising:

weighting a class estimate from a second artificial neural network based on whether the labelled crop is indicated as the background class.

3. The method of claim 1, further comprising:

determining a proportion of the object of interest that is included in the crop; and

comparing the determined proportion to at least one threshold value, and in which the labelling is based on the comparing.

4. A method of training an artificial neural network for image classification, comprising:

selecting a crop from an image;

determining whether the crop of the image includes an area surrounded by a predetermined bounding box;

discarding, prior to the training, the crop when the crop of the image does not include a portion of the area surrounded by the predetermined bounding box; and

training the artificial neural network to determine a label of the crop of the image including an object of interest with an indication of whether the object of interest is framed, partially framed, or not present in the crop, the artificial network trained for the image classification of the object of interest in the image with a plurality of crops from the image, the classification based on a class assigned to the labelled crop, each crop comprising at least a portion of the area surrounded by the predetermined bounding box.

5. The method of claim 4, further comprising generating a proposed bounding box for the image if the image lacks the predetermined bounding box, before discarding the crop.

6. The method of claim 5, further comprising:

selecting a new crop from the image;

determining whether the new crop of the image includes the area surrounded by the proposed bounding box; and

discarding the new crop if the new crop of the image does not include a portion of the area surrounded by the proposed bounding box.

7. The method of claim 5, in which the generating comprises:

localizing at least one object in the image; and

adding a border around the localized at least one object.

8. The method of claim 7, further comprising:

selecting a second crop including the proposed bounding box;

determining a proportion of the localized at least one object that is included in the second crop; and

updating the proposed bounding box based on the determined proportion.

9. An apparatus for image classification, comprising:

a memory; and

at least one processor coupled to the memory, the at least one processor configured;

to label, by a first artificial neural network, a crop from an image, including an object of interest with an indication of whether the object of interest is framed, partially framed, or not present in the crop;

to assign a fully framed class to the labelled crop, including the object of interest, when the object of interest is framed;

to assign a partially framed class to the labelled crop, including the object of interest, when the object of interest is partially framed;

to assign a background class to the labelled crop when the object of interest is not present in the crop; and

to classify the object of interest in the image based on a class assigned to the labelled crop.

10. The apparatus of claim 9, in which the at least one processor is further configured to weight a class estimate from a second artificial neural network, based on whether the labelled crop is indicated as the background class.

11. The apparatus of claim 9, in which the at least one processor is further configured:

to determine a proportion of the object of interest that is included in the crop; and

to compare the determined proportion to at least one threshold value, and in which the labelling is based on the comparing.

12. An apparatus for training an artificial neural network for image classification, comprising:

a memory; and

at least one processor coupled to the memory, the at least one processor configured;

to select a crop from an image;

to determine whether the crop of the image includes an area surrounded by a predetermined bounding box;

to discard, prior to the training, the crop when the crop of the image does not include a portion of the area surrounded by the predetermined bounding box; and

to train the artificial neural network to determine a label of the crop of the image including an object of interest with an indication of whether the object of interest is framed, partially framed, or not present in the crop, the artificial network trained for the image classification of the object of interest in the image with a plurality of crops from the image, the classification based on a class assigned to the labelled crop, each crop comprising at least a portion of the area surrounded by the predetermined bounding box.

13. The apparatus of claim 12, in which the at least one processor is further configured to generate a proposed bounding box for the image if the image lacks the predetermined bounding box, before discarding the crop.

14. The apparatus of claim 13, in which the at least one processor is further configured:

to select a new crop from the image;

to determine whether the new crop of the image includes the area surrounded by the proposed bounding box; and

to discard the new crop if the new crop of the image does not include a portion of the area surrounded by the proposed bounding box.

15. The apparatus of claim 13 in which the at least one processor is further configured to generate the proposed bounding box:

by localizing at least one object in the image; and

by adding a border around the localized at least one object.

16. The apparatus of claim 15, in which the at least one processor is further configured:

to select a second crop including the proposed bounding box;

to determine a proportion of the localized at least one object that is included in the second crop; and

to update the proposed bounding box based on the determined proportion.