Object-centric spatial pooling for image classification

• US 8,761,510 B2
• Filed: 11/14/2012
• Issued: 06/24/2014
• Est. Priority Date: 11/19/2011
• Status: Expired due to Fees

• Current Assignee: NEC Corporation
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Lin, Yuanqing, Yu, Kai, Russakovsky, Olga, Li, Fei-Fei
• Primary Examiner(s): LE, BRIAN Q
• Application Number:
  US13/676,494
• Publication Number:
  US 20130129199A1
• Time in Patent Office: 587 Days
• Field of Search: 382/115, 382/117, 382/118, 382/124, 382/125, 382/159, 382/173, 382/176, 382/181, 382/224, 382/225, 382/228
• US Class Current: 382/173
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

A method is provided for classifying an image. The method includes inferring location information of an object of interest in an input representation of the image. The method further includes determining foreground object features and background object features from the input representation of the image. The method additionally includes pooling the foreground object features separately from the background object features using the location information to form a new representation of the image. The new representation is different than the input representation of the image. The method also includes classifying the image based on the new representation of the image.

Claims

1. A method for classifying an image, comprising:

inferring location information of an object of interest in an input representation of the image;

determining foreground object features and background object features from the input representation of the image;

pooling the foreground object features separately from the background object features using the location information to form a new representation of the image, the new representation being different than the input representation of the image;

classifying the image based on the new representation of the image;

performing an outer iterative loop, the outer iterative loop comprising;

initializing a background region from the input representation of the image; and

incrementally shrinking at least a smallest bounding box from among a plurality of varying sized bounding boxes applied to the background region to incrementally increase a size of the background region with respect to at least the smallest bounding box;

performing an inner iterative loop that cooperates with the outer iterative loop, the inner iterative loop comprising;

inferring the location information of the object of interest from a current version of the smallest bounding box; and

from among a set of positive bounding boxes from positive images that are known to include the object of interest therein and from among a set of negative bounding boxes from negative images that are known to omit the object of interest therein, training a support vector machine classifier to discriminate the positive bounding boxes from the negative bounding boxes to refine the location information of the object of interest,

wherein said classifying step classifies the image based on discrimination results applied to a binary label of the image.

2. The method of claim 1, further comprising separating the foreground object features from the background object features using only image-level class labels.

3. The method of claim 2, wherein the image-level class levels are inferred using one or more bounding boxes.

4. The method of claim 2, further comprising training an object classification model using the foreground object features, the background object features, and the image-level class labels.

5. The method of claim 1, wherein said pooling and classifying steps combine the location information with classifying information in a joint model for both localization and classification.

6. The method of claim 1, wherein at least the smallest bounding box is incrementally shrunk until a predetermined condition is reached.

7. The method of claim 6, wherein the predetermined condition comprises at least the smallest bounding box entirely encompassing the object of interest including salient and non-salient portions thereof.

8. The method of claim 1, wherein at least the smallest bounding box is positioned around a most prominent instance of an object class within the background region, the most prominent instant of the object class being equated to the object of interest.

9. The method of claim 1, wherein the support vector machine classifier uses only features in the set of positive bounding boxes to the exclusion of features from an entirety of the image that are outside the positive bounding boxes.

10. The method of claim 1, further comprising retraining an object classification model initially used to separate the foreground object features from the background object features based on the new representation of the image.

11. The method of claim 10, further comprising updating a hypothesis of the location information of the object using the retrained object classification model, and repeating said determining and pooling steps to form another new representation of the Image.