System and method providing improved head motion estimations for animation

• US 7,706,575 B2
• Filed: 08/04/2004
• Issued: 04/27/2010
• Est. Priority Date: 12/06/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng
• Primary Examiner(s): Seth; Manav
• Application Number:
  US10/912,279
• Publication Number:
  US 20050008196A1
• Time in Patent Office: 2,092 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/115, 382/117, 382/118, 382/154, 382/181, 734/88, 348/154, 348/155, 356/27
• US Class Current: 382/118
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

The system provides improved procedures to estimate head motion between two images of a face. Locations of a number of distinct facial features are identified in two images. The identified locations can correspond to the eye corners, mouth corners and nose tip. The locations are converted into as a set of physical face parameters based on the symmetry of the identified distinct facial features. The set of physical parameters reduces the number of unknowns as compared to the number of equations used to determine the unknowns. An initial head motion estimate is determined by: (a) estimating each of the set of physical parameters, (b) estimating a first head pose transform corresponding to the first image, and (c) estimating a second head pose transform corresponding to the second image.

The head motion estimate can be incorporated into a feature matching algorithm to refine the head motion estimation and the physical facial parameters.

In one implementation, an inequality constraint is placed on a particular physical parameter—such as a nose tip, such that the parameter is constrained within a predetermined minimum and maximum value. The inequality constraint is converted to an equality constraint by using a penalty function. Then, the inequality constraint is used during the initial head motion estimation to add additional robustness to the motion estimation.

Claims

1. A computer-readable medium storing computer-executable instructions that, when executed on a computer, perform a method to estimate head motion between two images of a face, the method comprising operations of:

identifying, on the computer, locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting, on the computer, the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns; and

estimating, on the computer, head motion from identified points in the two images, the identified points being based on the set of physical face parameters.

2. One or more computer-readable media as recited in claim 1, wherein the operation of estimating head motion further comprises:

calculating, on the computer, an estimation objective function comprising a set of terms to estimate;

(a) each of the set of physical face parameters, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image.

3. One or more computer-readable media as recited in claim 1, further comprising after the operation of converting and before the operation of estimating:

adding, on the computer, an inequality constraint on a particular face parameter of the physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value.

4. One or more computer-readable media as recited in claim 3 further comprising:

wherein the particular face parameter corresponds to the nose tip; and

wherein the predetermined minimum value is zero (0) and the predetermined maximum value is a value based on absolute values of other of the locations.

5. One or more computer-readable media as recited in claim 3, further comprising converting, on the computer, the inequality constraint to an equality constraint using a penalty function.

6. One or more computer-readable media as recited in claim 1, wherein the operation of estimating head motion further comprises:

calculating, on the computer, an estimation objective function to determine an initial estimate of head rotation, the estimation objective function comprising a set of terms to estimate;

(a) each of the set of physical face parameters, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image;

placing, on the computer, an inequality constraint on a particular face parameter of the set of physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting, on the computer, the inequality constraint to an equality constraint using a penalty function; and

adding, on the computer, the equality constraint to the initial estimate.

7. One or more computer-readable media as recited in claim 1, wherein the identified locations correspond to the eye corners, mouth corners and nose tip.

8. One or more computer-readable media as recited in claim 2, wherein in the operation of estimating the head motion further comprises multiplying, on the computer, each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

9. One or more computer-readable media as recited in claim 5, wherein in the operation of providing further comprises:

multiplying, on the computer, each term of the estimation objective function and the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

10. One or more computer-readable media as recited in claim 1, wherein the identifying comprises accepting input from a human user.

11. A computer-readable medium storing computer-executable instructions that, when executed on a computer perform a method to estimate head motion between two images of a face, the method comprising operations of:

identifying, on the computer, locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting, on the computer, the locations into set of physical face parameters based on the symmetry of the identified distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns;

determining, on the computer, an initial estimation of head motion between the two images using the set of physical face parameters; and

refining, on the computer, the estimation of head motion by incorporating the initial estimation into a feature matching algorithm.

12. One or more computer-readable media as recited in claim 11, wherein the identified locations correspond to the eye corners, mouth corners and nose tip.

13. One or more computer-readable media as recited in claim 11, wherein the identifying comprises accepting input from a human user.

14. One or more computer-readable media as recited in claim 11, further comprising multiplying, on the computer, the refined head motion estimate by a confidence factor that indicates that each of the identified locations was identified with a same level of accuracy.

15. One or more computer-readable media as recited in claim 11, wherein determining an initial estimate of head motion comprises calculating, on the computer, an estimation objective function comprising a number of terms to estimate each of:

(a) the set of coordinates, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image.

16. One or more computer-readable media as recited in claim 15, wherein in the operation of determining further comprises multiplying, on the computer, each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

17. One or more computer-readable media as recited in claim 11, after the converting and before the determining, further comprising:

adding, on the computer, an inequality constraint on a particular parameter of the physical face parameters, such that the parameter is constrained within a predetermined minimum and maximum value.

18. One or more computer-readable media as recited in claim 17:

wherein the particular parameter corresponds to the nose tip; and

wherein the predetermined minimum value is zero (0) and the predetermined maximum value is a value based on absolute values of other of the locations.

19. One or more computer-readable media as recited in claim 17, further comprising converting, on the computer, the inequality constraint to an equality constraint using a penalty function.

20. One or more computer-readable media as recited in claim 11, wherein the determining further comprises:

calculating, on the computer, an estimation objective function to comprising a set of terms to estimate each of;

(a) the physical face parameters, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image;

adding, on the computer, an inequality constraint on a particular parameter of the physical face parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value;

converting, on the computer, the inequality constraint to an equality constraint using a penalty function; and

adding, on the computer, the equality constraint to the initial estimate.

21. One or more computer-readable media as recited in claim 20, wherein in the operation of providing further comprises:

multiplying, on the computer, each term of the estimation objective function and the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

22. A computer-readable medium storing computer-executable instructions that, when executed on a computer, performs a method to estimate motion between two images, the method comprising operations of:

identifying, on the computer, locations of a plurality of distinct features in the two images, the locations corresponding to symmetrical features shared between the two images, the locations corresponding to a number of unknowns determined upon estimation of motion between the two images;

converting, on the computer, the locations into a set of parameters based on the symmetry of the identified distinct features, the parameters reducing the number of unknowns as compared to a number of equations used to determine the unknowns; and

estimating, on the computer, motion between the two images using the parameters.

23. One or more computer-readable media as recited in claim 22, wherein the estimating further comprises:

providing, on the computer, an initial estimate of image rotation by estimation of each of the parameters, a first image pose transform corresponding to the first image, and a second image pose transform corresponding to the second image;

adding, on the computer, an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value;

converting, on the computer, the inequality constraint to an equality constraint using a penalty function; and

adding, on the computer, the equality constraint to the initial estimate.

24. One or more computer-readable media as recited in claim 22, wherein the estimating further comprises calculating, on the computer, an estimation objective function comprising a number of terms to estimate each of:

(a) the parameters, (b) a first image pose transform corresponding to the first image, and (c) a second image pose transform corresponding to the second image.

25. One or more computer-readable media as recited in claim 24, wherein in the operation of providing further comprises, on the computer, of multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of that corresponding data.

26. One or more computer-readable media as recited in claim 22, after the converting and before the estimating further comprising:

adding, on the computer, an inequality constraint on a particular parameter of the physical face parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value.

27. One or more computer-readable media as recited in claim 26, further comprising converting, on the computer, the inequality constraint to an equality constraint using a penalty function.

28. One or more computer-readable media as recited in claim 23, further comprising:

multiplying, on the computer, each term of a set of terms used to determine the initial estimate by a weighted contribution factor based on the confidence of data corresponding to the initial estimate.