LEARNING PART-BASED MODELS OF OBJECTS

• US 20130279800A1
• Filed: 04/23/2012
• Published: 10/24/2013
• Est. Priority Date: 04/23/2012
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ranganathan, Ananth, Mottaghi, Roozbeh
• Granted Patent: US 9,111,173 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations

Abstract

A system and method are disclosed for learning part-based object models during a learning phase from training images and applying the learned object models to an input image during runtime. The learned part-based object models are augmented by appearance-based models of the objects. The part-based object models correspond to the shapes of the parts of an object. The appearance-based models provide additional appearance cues to the object models for object classification. The approach to learning part-based object models has the capability of learning object models without using viewpoint labels of the objects. The learning is also invariant to scale and in-plane rotation of the objects.

Claims

1. A computer implemented method for learning part-based object models from a set of digital representations of images, the method comprising:

receiving a set of digital representations of images, each image having at least one object;

for each image;

extracting one or more shape features and appearance features from the image;

generating one or more shape models of the object based on the shape features, a shape model corresponding to a part of the object;

computing an appearance model for each shape model of the object based on the appearance features;

selecting one or more shape models and appearance models as reference shape models and appearance models of the object; and

storing the reference shape models and appearance models of the images.

2. The method of claim 1, wherein generating a shape model of the object in an image comprises;

generating one or more histogram of gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on one or more grouping criteria.

3. The method of claim 2, wherein the grouping criteria comprise at least one of following:

distance between two HOG cells, wherein the distance between the HOG cells is measured by a distance function; and

similarity of orientation between two HOG cell, wherein the similarity of orientation is compared based on the maximum magnitude of orientations of the two HOG cells.

4. The method of claim 2, wherein a HOG bundle is described by position of the bundle, orientation of the bundle, height and width of the bundle.

5. The method of claim 1, wherein selecting one or more shape models of the object comprises:

computing probability distributions of the shape models of the object; and

estimating likelihood of each shape model to be selected as a reference shape model of the object.

6. The method of claim 1, wherein selecting one or more shape models of the object further comprises:

selecting a first shape model from the shape models of the object as a root shape model of the object;

selecting a second shape model from the shape models of the object;

applying one or more learning rules to the second shape model;

determining whether the first shape model and the second shape model provide a better description of data for classifying the object than a background model; and

generating a shape model comprising the root shape model and the second shape model for the object.

7. The method of claim 1, further comprising:

augmenting each shape model of the object by its corresponding appearance model.

8. The method of claim 7, wherein augmenting each shape model of the object by its corresponding appearance model comprises:

re-computing the probability distribution of the shape model based on the appearance features contained in the appearance model.

9. A computer system for learning part-based object models from a set of digital representations of images, the system comprising:

an image pre-processing module for;

receiving a set of digital representations of images, each image having at least one object; and

for each image;

extracting one or more shape features and appearance features from the image;

generating one or more shape models of the object based on the shape features, a shape model corresponding to a part of the object;

an appearance module for computing an appearance model for each shape model of the object based on the appearance features; and

a part-based object module for;

for each image, selecting one or more shape models and appearance models as reference shape models and appearance models of the object; and

storing the reference shape models and appearance models of the images.

10. The system of claim 9, wherein generating a shape model of the object in an image comprises;

generating one or more histogram of gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on one or more grouping criteria.

11. The system of claim 10, wherein the grouping criteria comprise at least one of following:

distance between two HOG cells, wherein the distance between the HOG cells is measured by a distance function; and

similarity of orientation between two HOG cell, wherein the similarity of orientation is compared based on the maximum magnitude of orientations of the two HOG cells.

12. The system of claim 10, wherein a HOG bundle is described by position of the bundle, orientation of the bundle, height and width of the bundle.

13. The system of claim 9, wherein selecting one or more shape models of the object comprises:

computing probability distributions of the shape models of the object; and

estimating likelihood of each shape model to be selected as a reference shape model of the object.

14. The system of claim 9, wherein selecting one or more shape models of the object further comprises:

selecting a first shape model from the shape models of the object as a root shape model of the object;

selecting a second shape model from the shape models of the object;

applying one or more learning rules to the second shape model;

determining whether the first shape model and the second shape model provide a better description of data for classifying the object than a background model; and

generating a shape model comprising the root shape model and the second shape model for the object.

15. The system of claim 9, further comprising:

augmenting each shape model of the object by its corresponding appearance model.

16. The system of claim 15, wherein augmenting each shape model of the object by its corresponding appearance model comprises:

re-computing the probability distribution of the shape model based on the appearance features contained in the appearance model.

17. A non-transitory computer-readable storage medium storing executable computer program code for learning part-based object models from a set of digital representations of images, the computer program code comprising instructions for:

receiving a set of digital representations of images, each image having at least one object;

for each image;

extracting one or more shape features and appearance features from the image;

generating one or more shape models of the object based on the shape features, a shape model corresponding to a part of the object;

computing an appearance model for each shape model of the object based on the appearance features;

selecting one or more shape models and appearance models as reference shape models and appearance models of the object; and

storing the reference shape models and appearance models of the images.

18. The computer-readable storage medium of claim 17, wherein the computer program code for generating a shape model of the object in an image comprises instructions for:

generating one or more histogram of gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on one or more grouping criteria.

19. The computer-readable storage medium of claim of 18, wherein the computer program for selecting one or more shape models of the object comprises instructions for:

computing probability distributions of the shape models of the object; and

estimating likelihood of each shape model to be selected as a reference shape model of the object.

20. The computer-readable storage medium of claim of 18, further comprising computer program code for:

augmenting each shape model of the object by its corresponding appearance model.

21. A computer implemented method for classifying an object contained in digital representation of an image, the method comprising:

receiving one or more reference shape models and appearance models;

generating one or more shape models of the object based on shape features extracted from the image, a shape model corresponding to a part of the object;

comparing the generated shape models with the reference shape models;

selecting one or more candidate shape models of the object based on the comparison;

augmenting each candidate shape model with a corresponding appearance model of the object; and

determining classification of the object based on the augmented candidate shape models.

22. The method of claim 21, wherein determining classification of the object comprises:

determining whether there is an object in the image based on the augmented candidate shape models; and

responsive to the determination that there is an object in the image, determining location and type of the object.

23. A computer system for classifying an object contained in digital representation of an image, the system comprising:

an inference module for;

receiving one or more reference shape models and appearance models;

generating one or more shape models of the object based on shape features extracted from the image, a shape model corresponding to a part of the object;

comparing the generated shape models with the reference shape models;

selecting one or more candidate shape models of the object based on the comparison;

augmenting each candidate shape model with a corresponding appearance model of the object; and

determining classification of the object based on the augmented candidate shape models.

24. The system of claim 23, wherein determining classification of the object comprises:

determining whether there is an object in the image based on the augmented candidate shape models; and

responsive to the determination that there is an object in the image, determining location and type of the object.

25. A non-transitory computer-readable storage medium storing executable computer program code for classifying an object contained in digital representation of an image, the computer program code comprising instructions for:

receiving one or more reference shape models and appearance models;

generating one or more shape models of the object based on shape features extracted from the image, a shape model corresponding to a part of the object;

comparing the generated shape models with the reference shape models;

selecting one or more candidate shape models of the object based on the comparison;

augmenting each candidate shape model with a corresponding appearance model of the object; and

determining classification of the object based on the augmented candidate shape models.

26. The computer-readable storage medium of claim 25, wherein the computer program code for determining classification of the object comprises instructions for:

determining whether there is an object in the image based on the augmented candidate shape models; and

responsive to the determination that there is an object in the image, determining location and type of the object.