Geometry-driven feature point-based image synthesis

• US 7,379,071 B2
• Filed: 10/14/2003
• Issued: 05/27/2008
• Est. Priority Date: 10/14/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Qingshan, Shum, Heung-Yeung, Liu, Zicheng, Guo, Baining
• Primary Examiner(s): Zimmerman; Mark
• Assistant Examiner(s): Xu; Kevin K
• Application Number:
  US10/684,773
• Publication Number:
  US 20050078124A1
• Time in Patent Office: 1,687 Days
• Field of Search: 345629-641, 382298-300
• US Class Current: 345/582
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering

Abstract

A method and system uses geometry-driven feature point analysis to synthesize images including for example facial expressions. Given the feature point positions (geometry) of an expression, the method automatically synthesizes the corresponding expression image, which has photorealistic and natural looking expression details.

Claims

1. A computer implemented method for rendering a single frame of a synthesized image, comprising:

generating a desired geometric component corresponding to a selected image for the single frame based on identified feature points from a set of representative images, where each image of the set has the identified feature points with a corresponding reference image geometric component, and wherein the geometric component is a dimensional vector of feature point positions; and

generating the selected image for the single frame from a composite of the set of representative images based on the desired geometric component;

wherein the selected image and each of the set of representative images comprises a plurality of subregions defined adjacent to each other in the single frame wherein adjacent subregions share a common boundary; and

wherein generating a desired geometric component is performed for each subregion, and wherein the composite of the set of representative images is based on the corresponding desired geometric component for each subregion, and the selected image includes a synthesized subregion for each subregion based on the composite by projecting the desired geometric component into the convex hull of the corresponding reference image geometric components to obtain a set of coefficient, where a coefficient is associated with a representative image, and wherein a texture of the corresponding representative image as a function of each of the associated coefficients and blending at least some boundaries between adjacent subregions of the selected image order to generate the selected image.

2. The method of claim 1 wherein at least one synthesized subregion is based on a quantity of a set of representatives different than another synthesized subregion.

3. The method of claim 1 wherein the feature points correspond to a two-dimensional image.

4. The method of claim 1 wherein the feature points correspond to a three-dimensional image.

5. The method of claim 1 wherein generating a desired geometric component corresponding to a selected image based on identified feature points on the image comprises ascertaining a position of at least one feature point from a change in position of another feature point.

6. The method of claim 5 wherein the feature points are grouped in sets, each set pertaining to a different region of the selected image, and wherein ascertaining a position of at least one feature point comprises ascertaining positions of feature points in a set from a change in position of a feature point in the set.

7. The method of claim 6 wherein ascertaining a position of at least one feature point comprises ascertaining positions of feature points in the set using principle components derived from analyzing positional changes of feature points in the set for the representative images through principle component analysis.

8. The method of claim 7 wherein ascertaining a position of at least one feature point comprises ascertaining positions of feature points in a first set from a change in position of at least one feature point in another set.

9. The method of claim 8 wherein the sets are hierarchical.

10. The method of claim 1 and further comprising:

monitoring feature points of a user; and

wherein generating a desired geometric component corresponding to a selected image includes generating the desired geometric component corresponding to a change in position of feature points of the user; and

wherein generating the selected image corresponds to the change in position of feature points of the user.

11. The method of claim 10 wherein generating a desired geometric component corresponding to a selected image based on identified feature points comprises ascertaining a position of at least one feature point from a change in position of another feature point.

12. The method of claim 1 and further comprising:

rendering an image with feature points identified thereon; and

receiving information indicative of a user moving a feature point.

13. The method of claim 12 wherein generating a desired geometric component corresponding to a selected image based on identified feature points comprises ascertaining a position of at least one feature point from a change in position of another feature point.

14. The method of claim 1 wherein the selected image comprises a selected expression.

15. The method of claim 14 wherein the selected image comprises a selected facial expression.

16. The method of claim 1 wherein the each of the set of representative images are aligned with respect to a reference image.

27. The method of claim 1 wherein the each geometric component is calculated based on an objective function that is defined by a constraint and a plurality of feature points.

28. The method of claim 27 wherein the objective function is a positive semi definite quadratic form and the constraints are linear.

29. The method of claim 1 wherein blending is performed by identifying intensity values for adjacent regions on either side of the at least some boundaries and calculating an intensity value for the at least some of the boundaries based on the intensity values.

17. A computer implemented method for rendering a single frame of a synthesized image based on feature points, comprising:

accessing a set of stored representatives of various images, wherein each image of the set of the images has the same corresponding feature points associated therewith and a corresponding reference image geometric component and wherein the geometric component is a dimensional vector of feature point positions;

ascertaining a position of at least one feature point from a change in position of another feature point based on a change in movement of the selected feature point and based on the set of stored representatives of various images; and

rendering a new image for the single frame with two or more feature points having changed position;

wherein the new image and each image in the set of stored representatives of various images comprises a plurality of subregions defined adjacent to each other wherein each subregion of the new image has associated therewith a desired geometric component, wherein adjacent subregions share a common boundary, and rendering the new image includes rendering a synthesized subregion for each subregion by projecting the desired geometric component into the convex hull of the corresponding reference image geometric components to obtain a set of coefficients, where a coefficient is associated with a representative image, and wherein a texture of the subregion of the new image is based on combining the corresponding texture of the corresponding representative image as a function of each of the associated coefficients and blending at least some boundaries between adjacent subregions in the new image.

18. The method of claim 17 wherein the feature points are grouped in sets, each set pertaining to a different region of the selected image, and wherein ascertaining a position of at least one feature point comprises ascertaining positions of feature points in a set from a change in position of one feature point in the set.

19. The method of claim 18 wherein ascertaining a position of at least one feature point comprises ascertaining positions of feature points in the set using principle components derived from analyzing positional changes of feature points in the set of stored representatives of various images through principle component analysis.

20. The method of claim 19 wherein ascertaining a position of at least one feature point comprises ascertaining positions of feature points in a first set from a change in position of at least one feature point in another set.

21. The method of claim 20 wherein the sets are hierarchical.

22. The method of claim 17 wherein at least one synthesized subregion is based on a quantity of a set of representatives different than another synthesized subregion.

23. The method of claim 17 and further comprising monitoring feature points of a user.

30. The method of claim 17 wherein blending is performed by identifying intensity values for adjacent regions on either side of the at least some boundaries and calculating an intensity value for the at least some of the boundaries based on the intensity values.

24. A computer implemented method for rendering a single frame of a synthesized image based on feature points, comprising:

rendering a image with identified feature points, wherein the feature points are grouped in hierarchical sets, each set pertaining to a different region of the image;

receiving information indicative of a user moving a selected feature point;

accessing a set of stored representatives of various images, wherein each image of the set of stored images has the same corresponding feature points associated therewith and a corresponding reference image geometric component, and wherein the geometric component is a dimensional vector of feature point positions;

ascertaining a position of at least one feature point in each of two different sets from a change in position of another feature point based on a change in movement of the selected feature point and based on the set of stored representatives of various images, wherein ascertaining comprises ascertaining positions of feature points in a set from a change in position of one feature point int he set using principle components derived from analyzing positional changes of feature points in the set of stored representatives of various images through principle component analysis; and

rendering a new image for the single frame with two or more feature points having changed position in two sets;

wherein the new image and each image in the set of stored representatives of various images comprises a plurality of subregions adjacent to each other wherein each subregion of the new image has associated therewith a desired geometric component, wherein adjacent subregions share a common boundary, and rendering a synthesized subregion for each subregion in the new image is obtained by protecting t he desired geometric component into the convex hull of t he corresponding reference image geometric components to obtain a set of coefficients, where a coefficient is associated with a representative image, and wherein a texture of the selected image is based on combining the corresponding texture of the corresponding representative image as a function of each of the associated coefficients.

25. The method of claim 24 wherein the wherein the image comprises a facial image.

26. The method of claim 24 wherein at least one synthesized subregion is based on a quantity of a set of representatives different than another synthesized subregion.