Learning part-based models of objects

• US 9,111,173 B2
• Filed: 04/23/2012
• Issued: 08/18/2015
• Est. Priority Date: 04/23/2012
• Status: Active Grant

• Current Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Ranganathan, Ananth, Mottaghi, Roozbeh
• Primary Examiner(s): Fitzpatrick, Atiba O
• Assistant Examiner(s): Potts, Ryan P
• Application Number:
  US13/453,571
• Publication Number:
  US 20130279800A1
• Time in Patent Office: 1,212 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations

Abstract

A system and method are provided for learning part-based object models during a learning phase from training images and applying the learned object models to an input image during runtime. The learned part-based object models are augmented by appearance-based models of the objects. The part-based object models correspond to the shapes of the parts of an object. The appearance-based models provide additional appearance cues to the object models for object classification. The approach to learning part-based object models has the capability of learning object models without using viewpoint labels of the objects. The learning is also invariant to scale and in-plane rotation of the objects.

Claims

1. A computer implemented method for learning part-based object models from a set of digital representations of images, the method comprising:

receiving a set of digital representations of images, each image having at least one object;

for each image;

extracting one or more shape features from the image;

extracting one or more appearance features from the image;

generating one or more shape models of the object based on the shape features, a shape model corresponding to a part of the object, generating a shape model of the object comprising;

generating one or more histogram of oriented gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells;

computing an appearance model for each shape model of the object based on the appearance features;

selecting one or more shape models as reference shape models of the object;

selecting one or more appearance models as reference appearance models of the object; and

storing the reference shape models of the images; and

storing the reference appearance models of the images.

2. The method of claim 1, wherein generating a shape model of the object further comprises:

grouping two or more HOG cells into a HOG bundle based on distance between two HOG cells, wherein the distance between the HOG cells is measured by a distance function.

3. The method of claim 1, wherein a HOG bundle is described by position of the bundle, orientation of the bundle, height and width of the bundle.

4. The method of claim 1, wherein selecting one or more shape models of the object comprises:

computing probability distributions of the shape models of the object; and

estimating likelihood of each shape model to be selected as a reference shape model of the object.

5. The method of claim 4, further comprising:

re-computing the probability distribution of each shape model based on the appearance features contained in a corresponding appearance model.

6. The method of claim 1, wherein selecting one or more shape models of the object further comprises:

selecting a first shape model from the shape models of the object as a root shape model of the object;

selecting a second shape model from the shape models of the object;

applying one or more learning rules to the second shape model;

determining whether the first shape model and the second shape model provide a better description of data for classifying the object than a background model; and

generating a shape model comprising the root shape model and the second shape model for the object.

7. The method of claim 1, wherein generating a shape model further comprises:

grouping two or more HOG cells into a HOG bundle based on distance between the HOG cells measured by a chi square distance function and similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells.

8. The method of claim 1, wherein generating a shape model further comprises:

comparing orientation of a HOG cell with a predefined orientation threshold; and

excluding the HOG cell from grouping with other HOG cells responsive to the evaluation of the difference between the orientation of the HOG cell and the predefined orientation threshold.

9. A computer system for learning part-based object models from a set of digital representations of images, the system comprising:

a non-transitory computer-readable storage medium storing executable computer program instructions; and

a computer processor executing the computer program instructions to perform steps comprising;

receiving a set of digital representations of images, each image having at least one object; and

for each image;

extracting one or more shape features from the image;

extracting one or more appearance features from the image;

generating one or more shape models of the object based on the shape features, a shape model corresponding to a part of the object, generating a shape model comprising;

generating one or more histogram of oriented gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells;

computing an appearance model for each shape model of the object based on the appearance features; and

selecting one or more shape models as reference shape models of the object;

selecting one or more appearance models as reference appearance models of the object; and

storing the reference shape models of the images; and

storing the reference appearance models of the images.

10. The system of claim 9, wherein generating a shape model of the object further comprises:

grouping two or more HOG cells into a HOG bundle based on distance between two HOG cells, wherein the distance between the HOG cells is measured by a distance function.

11. The system of claim 9, wherein a HOG bundle is described by position of the bundle, orientation of the bundle, height and width of the bundle.

12. The system of claim 9, wherein selecting one or more shape models of the object comprises:

computing probability distributions of the shape models of the object; and

estimating likelihood of each shape model to be selected as a reference shape model of the object.

13. The system of claim 12, further comprising:

re-computing the probability distribution of each shape model based on the appearance features contained in a corresponding appearance model.

14. The system of claim 9, wherein selecting one or more shape models of the object further comprises:

selecting a first shape model from the shape models of the object as a root shape model of the object;

selecting a second shape model from the shape models of the object;

applying one or more learning rules to the second shape model;

determining whether the first shape model and the second shape model provide a better description of data for classifying the object than a background model; and

generating a shape model comprising the root shape model and the second shape model for the object.

15. The system of claim 9, wherein generating a shape model further comprises:

grouping two or more HOG cells into a HOG bundle based on distance between the HOG cells measured by a chi square distance function and similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells.

16. The system of claim 9, wherein generating a shape model further comprises:

comparing orientation of a HOG cell with a predefined orientation threshold; and

excluding the HOG cell from grouping with other HOG cells responsive to the evaluation of the difference between the orientation of the HOG cell and the predefined orientation threshold.

17. A non-transitory computer-readable storage medium storing executable computer program instructions for learning part-based object models from a set of digital representations of images, the computer program instructions comprising instructions for:

receiving a set of digital representations of images, each image having at least one object;

for each image;

extracting one or more shape features from the image;

extracting one or more appearance features from the image;

generating one or more shape models of the object based on the shape features, a shape model corresponding to a part of the object, generating one or more shape models comprising;

generating one or more histogram of oriented gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells;

computing an appearance model for each shape model of the object based on the appearance features;

selecting one or more shape models as reference shape models of the object; and

selecting one or more appearance models as reference appearance models of the object; and

storing the reference shape models and appearance models of the images; and

storing the reference appearance models of the images.

18. The computer-readable storage medium of claim of 17, wherein the computer program instructions for selecting one or more shape models of the object comprise instructions for:

computing probability distributions of the shape models of the object;

estimating likelihood of each shape model to be selected as a reference shape model of the object; and

re-computing the probability distribution of each shape model based on the appearance features contained in a corresponding appearance model.

19. The computer-readable storage medium of claim 17, wherein the computer program instructions for generating a shape model further comprise instructions for:

grouping two or more HOG cells into a HOG bundle based on distance between the HOG cells measured by a chi square distance function and similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells.

20. The computer-readable storage medium of claim 17, wherein the computer program instructions for generating a shape model further comprise instructions for:

comparing orientation of a HOG cell with a predefined orientation threshold; and

excluding the HOG cell from grouping with other HOG cells responsive to the evaluation of the difference between the orientation of the HOG cell and the predefined orientation threshold.

21. A computer implemented method for classifying an object contained in digital representation of an image, the method comprising:

receiving one or more reference shape models;

receiving one or more reference appearance models;

generating one or more shape models of the object based on shape features extracted from the image, a shape model corresponding to a part of the object, generating a shape model comprising;

generating one or more histogram of oriented gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells;

comparing the generated shape models with the reference shape models;

selecting one or more candidate shape models of the object based on the comparison;

augmenting each candidate shape model with a corresponding appearance model of the object; and

determining classification of the object based on the augmented candidate shape models.

22. The method of claim 21, wherein determining classification of the object comprises:

determining whether there is an object in the image based on the augmented candidate shape models; and

responsive to the determination that there is an object in the image, determining location and type of the object.

23. A computer system for classifying an object contained in digital representation of an image, the system comprising:

a non-transitory computer-readable storage medium storing executable computer program instructions that when executed by a computer processor cause the computer processor to;

receive one or more reference shape models;

receive one or more reference appearance models;

generate one or more shape models of the object based on shape features extracted from the image, a shape model corresponding to a part of the object, generating a shape model comprising;

generating one or more histogram of oriented gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells;

compare the generated shape models with the reference shape models;

select one or more candidate shape models of the object based on the comparison;

augment each candidate shape model with a corresponding appearance model of the object; and

determine classification of the object based on the augmented candidate shape models.

24. The system of claim 23, wherein determining classification of the object comprises:

determining whether there is an object in the image based on the augmented candidate shape models; and

responsive to the determination that there is an object in the image, determining location and type of the object.

25. A non-transitory computer-readable storage medium storing executable computer program instructions for classifying an object contained in digital representation of an image, the computer program instructions comprising instructions for:

receiving one or more reference shape models;

receiving one or more reference appearance models;

generating one or more shape models of the object based on shape features extracted from the image, a shape model corresponding to a part of the object, generating one or more shape models comprising;

generating one or more histogram of oriented gradient (HOG) cells for a number of pixel of the image; and

grouping two or more HOG cells into a HOG bundle based on similarity of orientations of the HOG cells with respect to the maximum magnitude of orientations of the HOG cells;

comparing the generated shape models with the reference shape models;

selecting one or more candidate shape models of the object based on the comparison;

augmenting each candidate shape model with a corresponding appearance model of the object; and

determining classification of the object based on the augmented candidate shape models.

26. The computer-readable storage medium of claim 25, wherein the computer program instructions for determining classification of the object comprise instructions for:

determining whether there is an object in the image based on the augmented candidate shape models; and

responsive to the determination that there is an object in the image, determining location and type of the object.