Model-based object image processing

• US 8,131,063 B2
• Filed: 02/25/2009
• Issued: 03/06/2012
• Est. Priority Date: 07/16/2008
• Status: Expired due to Fees

• Current Assignee: 138 East LCD Advancements Limited (Vector Capital Corporation)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Xiao, Jing, Shiell, Derek
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US12/392,808
• Publication Number:
  US 20100013832A1
• Time in Patent Office: 1,105 Days
• Field of Search: 382/118, 382/154, 382/276, 382/285, 345/6, 345/419, 345/420, 345/422, 345/427, 345/621, 345/630, 345/653, 345/654, 345/664, 700/98, 700/163, 700/182, 702/152, 702/153, 715/225, 715/771, 715/852, 715/964, 715/970
• US Class Current: 382/154
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Aspects of the present invention include systems and methods for forming generative models, for utilizing those models, or both. In embodiments, an object model fitting system can be developed comprising a 3D active appearance model (AAM) model. The 3D AAM comprises an appearance model comprising a set of subcomponent appearance models that is constrained by a 3D shape model. In embodiments, the 3D AAM may be generated using a balanced set of training images. The object model fitting system may further comprise one or more manifold constraints, one or more weighting factors, or both. Applications of the present invention include, but are not limited to, modeling and/or fitting face images, although the teachings of the present invention can be applied to modeling/fitting other objects.

Claims

1. A computer-readable medium carrying one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to fit an input image to an object model by performing the steps comprising:

[a] using a object model fitting system comprising a set of weights, at least one set of manifold constraints, and a three-dimensional (3D) object active appearance model (AAM) comprising a 3D object shape model that receives shape parameter values and pose parameter values and an object appearance model comprising a set of object subcomponent appearance models that receive appearance parameter values, to generate a first image and a second image,

the first image being generated using the 3D object shape model, the input image, the shape parameter values, and pose parameter values to generate the first image from the input image; and

the second image being generated using the object appearance model and appearance parameter values to generate the second image;

[b] calculating a set of fitting errors representing a difference between the first image and the second image;

[c] responsive to a stop condition not being satisfied;

updating at least some of weights in the set of weights using at least some of the shape and pose parameter values;

updating at least some of the shape, pose, and appearance parameter values using at least some of the set of errors and the at least one set of manifold constraints; and

returning to step [a].

2. The computer-readable medium of claim 1 wherein the input image is an image comprising a face and the 3D object AAM models shape and appearance deformations across faces.

3. The computer-readable medium of claim 1 wherein the set of fitting errors comprise registration errors, and fitting the input image to the 3D object AAM is performed by an inverse compositional method comprising finding an optimal 3D shape parameter update.

4. The computer-readable medium of claim 1 wherein the step of updating at least some of weights in the set of weights using at least some of the shape and pose parameter values comprises:

dividing the first image into a set of subdivisions, wherein a subdivision comprises one or more pixels;

using the at least some of the shape and pose parameter values to determine angles for at least some of the subdivision in the first image, an angle being determined as the angle between a normal vector from a subdivision in the first image and a camera axis to the subdivision; and

updating at least some of the set of weights in relation to at least some of the angles.

5. The computer-readable medium of claim 4 wherein updating at least some of the set of weights in relation to at least some of the angles comprises assigning a weight to a region, the weight being inversely related to the size of an angle associated with that region.

6. The computer-readable medium of claim 5 wherein the region is a subdivision.

7. The computer-readable medium of claim 5 wherein the region is formed using one or more subdivisions and the angle associated with the region is obtained using the angles associated with the one or more subdivisions.

8. The computer-readable medium of claim 4 wherein the set of errors comprises registration errors and updating at least some of weights in the set of weights further comprises using at least some of the registration errors to update the set of weights.

9. The computer-readable medium of claim 1 wherein the at least one set of manifold constraints are generated using an appearance manifold formed using appearance parameters associated with the object appearance model and using a shape manifold formed using shape parameters associated with the 3D object shape model.

10. The computer-readable medium of claim 9 wherein a manifold constraint is associated with a distance of a model parameter vector instance to a cluster center within a manifold, the manifold comprising a set of model parameter vector instances obtained by fitting a training set of images to an interim object AAM comprising the 3D object shape model and the object appearance model.

11. The computer-readable medium of claim 10 wherein a cluster analysis is applied to the set of model parameter vector instances to obtain a set of cluster centers and a set of cluster co-variances associated with the set of cluster centers, and the cluster center is from the set of cluster centers.

12. A computer-readable medium carrying one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to form an object model fitting system based on a set of training images by performing the steps comprising:

[a] generating a balanced object model by;

[a1] generating a three-dimensional (3D) object shape model comprising shape parameters, the 3D object shape model based on the set of training images;

[a2] generating a object appearance model comprising a linear combination of a set of subcomponent object appearance models comprising texture parameters, the object appearance model based on the set of training images;

[a3] forming an interim object model comprising the 3D object shape model and the object appearance model;

[a4] for each object image in a set of images selected from the set of training images, generating an object model estimate by fitting the object image to the interim object model, and calculating a fitting error between the object image and the object model estimate; and

[a5] responsive to identifying a subset of training images having associated fitting errors that are larger than an error threshold value, balancing the set of training images based on the associated fitting errors and returning to step [a1]; and

[a5] responsive to identifying no subset of training images having associated fitting errors that are larger than an error threshold value, setting the interim object model as the balanced object model;

and

using the balanced object model to generate at least one set of manifold constraints associated with a manifold, the manifold based on a vector of a type of model parameters, the type of model parameters comprising shape and texture; and

forming the object model fitting system comprising the balanced object model and the at least one set of manifold constraints.

13. The computer-readable medium of claim 12 wherein using the balanced object model to generate a least one set of manifold constraints associated with a manifold comprises:

obtaining a set of model parameter vector instances by;

for each image from a set of images selected from the set of training images, fitting an image to the balanced object model to obtain a reconstructed object image instance;

extracting a model parameter vector instance from the reconstructed object image instance; and

adding the model parameter vector instance to the set of model parameter vector instances;

forming the manifold using the set of model parameter vector instances; and

generating a set of manifold constraints by applying a cluster analysis to the set of model parameter vector instances.

14. The computer-readable medium of claim 13 wherein generating the set of manifold constraints is based on a set of cluster centers and a set of cluster co-variances associated with the set of cluster centers.

15. The computer-readable medium of claim 14 wherein the cluster analysis is K-Means clustering.

16. The computer-readable medium of claim 12 wherein balancing the set of training images comprises increasing the size of the set of training images by adding a second set of images to the set of training images, each of the second set of images having at least some of the characteristics of the subset of training images.

17. The computer-readable medium of claim 16 wherein balancing the set of training images comprises weighting at least some of the images within the set of training images.

18. The computer-readable medium of claim 17 wherein weighting at least some of the images comprises assigning weights in relation to fitting errors associated with the at least some of the images.

19. A system for forming a reconstructed object image from an input image comprising pixels, the system comprising:

an object model fitter, coupled to receive the input image, that forms the reconstructed object image by fitting the input image to an object fitting system comprising a set of weights, at least one set of manifold constraints, and a three-dimensional (3D) object active appearance model (AAM) comprising a 3D object shape model that receives shape parameter values and pose parameter values and an object appearance model comprising a set of object subcomponent appearance models that receive appearance parameter values, the object model fitter comprising;

a reconstructed object estimator, coupled to receive the input image, the set of weights, the 3D AAM, 3D shape parameter values, pose parameter values, and appearance parameter values, that generates a first image using the 3D object shape model, the input image, the shape parameter values, and the pose parameter values, and that generates a second image obtained using the object appearance model, at least some of the set of weights, and the appearance parameter values; and

a reconstructed object updater, coupled to receive the first and second images, that calculates a set of fitting errors representing a difference between the first and second images, and updates at least some of weights in the set of weights using at least some of the pose parameter values, and updates at least some of the shape, pose, and appearance parameter values using at least some of the set of errors and the at least one set of manifold constraints.

20. The system of claim 19 further comprising an object model builder, coupled to receive a set of training images, that forms the 3D AAM and at least one manifold based on the set of training images.

21. The system of claim 20 wherein the object model builder comprises:

a 3D shape reconstructor and modeler, coupled to receive a set of shapes obtained from a set of labeled feature points in the set of training images, that forms the 3D object shape model based on the set of shapes;

a subcomponent appearance modeler, coupled to receive the set of training images, that forms the set of object subcomponent appearance models using the set of training images; and

a manifold builder, coupled to receive instances of at least one parameter type obtained using the 3D AAM and at least some of the set of training images, that builds a manifold using the instances of the at least one parameter type.