Method and System For Localizing Parts of an Object in an Image For Computer Vision Applications

• US 20120308124A1
• Filed: 06/04/2012
• Published: 12/06/2012
• Est. Priority Date: 06/02/2011
• Status: Active Grant

• Current Assignee: Kriegman-Belhumeur Vision Technologies LLC (Dropbox, Inc.)
• Original Assignee: Kriegman-Belhumeur Vision Technologies LLC (Dropbox, Inc.)
• Inventors: Belhumeur, Peter N., Jacobs, David W., Kriegman, David J., Kumar, Neeraj
• Granted Patent: US 8,811,726 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 18/24155 Bayesian classification
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A method is provided for localizing parts of an object in an image by training local detectors using labeled image exemplars with fiducial points corresponding to parts within the image. Each local detector generates a detector score corresponding to the likelihood that a desired part is located at a given location within the image exemplar. A non-parametric global model of the locations of the fiducial points is generated for each of at least a portion of the image exemplars. An input image is analyzed using the trained local detectors, and a Bayesian objective function is derived for the input image from the non-parametric model and detector scores. The Bayesian objective function is optimized using a consensus of global models, and an output is generated with locations of the fiducial points labeled within the object in the image.

Claims

1. A method for localizing parts of an object in an input image, comprising:

training a plurality of local detectors using a plurality of image exemplars as training images, wherein each image exemplar is labeled with a plurality of fiducial points, and wherein each local detector generates a detector score when applied at a location in a training image corresponding to a likelihood that a desired part is located at the location within the training image;

generating a plurality of non-parametric global models using at least a portion of the plurality of image exemplars;

inputting data corresponding to the input image;

applying the trained local detectors to the input image to generate detector scores for the input image;

deriving a Bayesian objective function from the plurality of non-parametric global models and the detector scores for the input image using an assumption that locations of fiducial points within the input image are represented within its corresponding global model as hidden variables;

optimizing the Bayesian objective function to obtain a consensus set of global models for the hidden variables that best fits the data corresponding to the input image; and

generating an output comprising locations of the fiducial points within the object in the input image.

2. The method of claim 1 wherein the object is a face.

3. The method of claim 1, wherein the step of optimizing the Bayesian objective function comprises generating a plurality of plausible values for the hidden variables by:

selecting a random image exemplar from the plurality of image exemplars;

selecting from within the random image exemplar at least two random fiducial points;

applying a similarity transform to align the at least two random fiducial points with peaks of the detector output;

evaluating a fit of the random image exemplar and similarity transform to the data; and

repeating the steps of selecting a random image exemplar, selecting at least two random fiducial points, applying a similarity transform and evaluating a fit for a plurality of iterations until a desired fit to the data is obtained.

4. The method of claim 1, wherein the plurality of local detectors comprise sliding window detectors.

5. The method of claim 4, wherein the sliding window detectors comprise support vector machines.

6. The method of claim 4, wherein the sliding window detectors use features comprising scale invariant feature transform descriptors.

7. The method of claim 1, wherein generating the output comprises displaying on a monitor or printout the image of the object with markings indicating the fiducial points within the image.

8. The method of claim 1, wherein generating the output comprises storing the output in a memory for further processing by an image recognition system.

9. The method of claim 1, wherein the input image comprises features in addition to the object and further comprising pre-processing the input image to select and extract an area within the input image which contains only the object.

10. A method for localizing parts of an object in an input image, comprising:

training a plurality of local detectors using at least a portion of a plurality of image exemplars as training images, wherein each image exemplar is labeled with fiducial points corresponding to parts within the image, and wherein each local detector generates a detector score when applied at one location of a plurality of locations of fiducial points in the training images corresponding to a likelihood that a desired part is located at the location within the training image;

generating a non-parametric model of the plurality of locations of the fiducial points in each of at least a portion of the plurality of image exemplars;

inputting data corresponding to the input image;

applying the trained local detectors to the input image to generate detector scores for the input image;

deriving a Bayesian objective function for the input image from the non-parametric model and detector scores; and

generating an output comprising locations of the fiducial points within the object in the image.

11. The method of claim 10 wherein the object is a face.

12. The method of claim 10, wherein the step of deriving a Bayesian objective function comprises:

using an assumption that locations of fiducial points within the image exemplar are represented within its corresponding global model as hidden variables; and

optimizing the Bayesian objective function to obtain a consensus set of global models for the hidden variables that best fits the data corresponding to the image.

13. The method of claim 12, wherein the step of optimizing the Bayesian objective function comprises generating a plurality of plausible values for the hidden variables by:

selecting a random image exemplar from the plurality of image exemplars;

selecting from within the random image exemplar at least two random fiducial points;

applying a similarity transform to align the at least two random fiducial points with peaks of the detector output;

evaluating a fit of the random image exemplar and similarity transform to the data; and

repeating the steps of selecting a random image exemplar, selecting at least two random fiducial points, applying a similarity transform and evaluating a fit for a plurality of iterations until a desired fit to the data is obtained.

14. The method of claim 10, wherein the plurality of local detectors comprise sliding window detectors.

15. The method of claim 14, wherein the sliding window detectors comprise support vector machines.

16. The method of claim 14, wherein the sliding window detectors use features comprising scale invariant feature transform descriptors.

17. The method of claim 10, wherein generating the output comprises displaying on a monitor or printout the image of the object with markings indicating the fiducial points within the image.

18. The method of claim 10, wherein generating the output comprises storing the output in a memory for further processing by an image recognition system.

19. The method of claim 12, wherein the consensus set contains only one global model.

20. The method of claim 10, wherein the input image comprises features in addition to the object and further comprising pre-processing the input image to select and extract an area within the input image which contains only the object.

21. A computer-program product embodied on a non-transitory computer-readable medium comprising instructions for receiving a plurality of image exemplars, and further comprising instructions for:

training a plurality of local detectors using at least a portion of a plurality of image exemplars as training images, wherein each image exemplar is labeled with fiducial points corresponding to parts within the image, and wherein each local detector generates a detector score when applied at one location of a plurality of locations of fiducial points in the training images corresponding to a likelihood that a desired part is located at the location within the training image;

generating a non-parametric model of the plurality of locations of the fiducial points in each of at least a portion of the plurality of image exemplars;

inputting data corresponding to the input image;

applying the trained local detectors to the input image to generate detector scores for the input image;

deriving a Bayesian objective function for the input image from the non-parametric model and detector scores; and

generating an output comprising locations of the fiducial points within the object in the image.

22. The computer-program product of claim 21, wherein the step of deriving a Bayesian objective function comprises:

using an assumption that locations of fiducial points within the image exemplar are represented within its corresponding global model as hidden variables; and

optimizing the Bayesian objective function to obtain a consensus set of global models for the hidden variables that best fits the data corresponding to the image.

23. The computer-program product of claim 22, wherein the consensus set contains only one global model.

24. The computer-program product of claim 22, wherein the step of optimizing the Bayesian objective function comprises generating a plurality of plausible values for the hidden variables by:

selecting a random image exemplar from the plurality of image exemplars;

selecting from within the random image exemplar at least two random fiducial points;

applying a similarity transform to align the at least two random fiducial points with the peaks of the detector output;

evaluating a fit of the random image exemplar and similarity transform to the data; and

repeating the steps of selecting a random image exemplar, selecting at least two random fiducial points, applying a similarity transform and evaluating a fit for a plurality of iterations until a desired fit to the data is obtained

25. The computer-program product of claim 21, wherein the plurality of local detectors comprise sliding window detectors.

26. The computer-program product of claim 25, wherein the sliding window detectors comprise support vector machines.

27. The computer-program product of claim 25, wherein the sliding window detectors use features comprising scale invariant feature transform descriptors.

28. The computer-program product of claim 21, wherein generating the output comprises displaying on a monitor or printout the image of the object with markings indicating the fiducial points within the image.

29. The computer-program product of claim 21, wherein generating the output comprises storing the output in a memory for further processing by an image recognition system.

30. The computer-program product of claim 21, wherein the input image comprises features in addition to the object and further comprising pre-processing the input image to select and extract an area within the input image which contains only the object.

31. The computer-program product of claim 21, wherein the object is a face.