System and method providing improved head motion estimations for animation

• US 20050074145A1
• Filed: 12/01/2004
• Published: 04/07/2005
• Est. Priority Date: 12/06/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Liu, Zicheng
• Granted Patent: US 7,039,219 B2
• Time in Patent Office: Days
• US Class Current: 382/109
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

Systems and methods to estimate head motion between two images of a face are described. In one aspect, locations of a plurality of distinct facial features in the two images are identified. The locations correspond to a number of unknowns that are determined upon estimation of head motion. The number of unknowns are determined by a number of equations. The identified locations are converted into a set of physical face parameters based on the symmetry of the distinct facial features. The set of physical face parameters reduce the number of unknowns as compared to the number of equations used to determine the unknowns. An inequality constraint is added to a particular face parameter of the physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value. The inequality constraint is converted to an equality constraint using a penalty function. Head motion is estimated from identified points in the two images. The identified points are based on the set of physical face parameters.

Claims

1. A computing device to estimate head motion between two images of a face, the computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

identifying locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns;

adding an inequality constraint on a particular face parameter of the physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

estimating head motion from identified points in the two images, the identified points being based on the set of physical face parameters.

2. A computing device as recited in claim 1:

wherein the particular face parameter corresponds to the nose tip; and

wherein the predetermined minimum value is zero (0) and the predetermined maximum value is a value based on absolute values of other of the locations.

3. A computing device as recited in claim 1, wherein the identified locations correspond to eye corners, mouth corners and nose tip.

4. A computing device as recited in claim 1, wherein identifying the locations further comprises accepting input from a human user.

5. A computing device as recited in claim 1, wherein the computer-program instructions for estimating head motion further comprise instructions for calculating an estimation objective function comprising a set of terms to estimate:

(a) each of the set of physical face parameters;

(b) a first head pose transform corresponding to the first image; and

,

(c) a second head pose transform corresponding to the second image.

6. A computing device as recited in claim 5, wherein estimating the head motion further comprises multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

7. A computing device as recited in claim 5, wherein estimating the head motion further comprises multiplying each term of the estimation objective function and the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

8. A computer-readable medium comprising computer-program instructions executable by a processor to estimate head motion between two images of a face, the computer-program instructions comprising instructions for:

identifying locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns;

adding an inequality constraint on a particular face parameter of the physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

estimating head motion from identified points in the two images, the identified points being based on the set of physical face parameters.

9. A computer-readable medium as recited in claim 8:

wherein the particular face parameter corresponds to the nose tip; and

wherein the predetermined minimum value is zero (0) and the predetermined maximum value is a value based on absolute values of other of the locations.

10. A computer-readable medium as recited in claim 8, wherein the identified locations correspond to eye corners, mouth corners and nose tip.

11. A computer-readable medium as recited in claim 8, wherein identifying the locations further comprises accepting input from a human user.

12. A computer-readable medium as recited in claim 8, wherein the computer-program instructions for estimating head motion further comprise instructions for calculating an estimation objective function comprising a set of terms to estimate:

(a) each of the set of physical face parameters;

(b) a first head pose transform corresponding to the first image; and

(c) a second head pose transform corresponding to the second image.

13. A computer-readable medium as recited in claim 12, wherein estimating the head motion further comprises multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

14. A computer-readable medium as recited in claim 12, wherein estimating the head motion further comprises multiplying each term of the estimation objective function and the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

15. A computing device to estimate head motion between two images of a face, the computing device comprising:

identifying means to identify locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting means to convert the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns;

adding means to add an inequality constraint on a particular face parameter of the physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting means to change the inequality constraint to an equality constraint using a penalty function; and

estimating means to estimate head motion from identified points in the two images, the identified points being based on the set of physical face parameters.

16. A computing device as recited in claim 15, wherein the identifying means further comprises accepting means to accept input from a human user.

17. A computing device as recited in claim 15, wherein the estimating means further comprise calculating means to calculate an estimation objective function comprising a set of terms to estimate:

(a) each of the set of physical face parameters;

(b) a first head pose transform corresponding to the first image; and

(c) a second head pose transform corresponding to the second image.

18. A computing device as recited in claim 17, wherein the estimating means further comprise multiplying means to multiply each term of the estimation objective function and the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

19. A computing device to estimate head motion between two images of a face, the computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor for;

identifying locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns; and

estimating head motion from identified points in the two images, the identified points being based on the set of physical face parameters; and

wherein the operation of estimating head motion comprise operations of;

calculating an estimation objective function to determine an initial estimate of head rotation, the estimation objective function comprising a set of terms to estimate;

(a) each of the set of physical face parameters, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image;

placing an inequality constraint on a particular face parameter of the set of physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

adding the inequality constraint to the initial estimate.

20. A computer-readable medium comprising computer-program instructions to estimate head motion between two images of a face, the computer-program instructions comprising instructions for:

identifying locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns; and

estimating head motion from identified points in the two images, the identified points being based on the set of physical face parameters; and

wherein the operation of estimating head motion comprise operations of;

calculating an estimation objective function to determine an initial estimate of head rotation, the estimation objective function comprising a set of terms to estimate;

(a) each of the set of physical face parameters, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image;

placing an inequality constraint on a particular face parameter of the set of physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

adding the inequality constraint to the initial estimate.

21. A computing device to estimate head motion between two images of a face, the computing device comprising:

identifying means to identify locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting means to convert the identified locations into a set of physical face parameters based on the symmetry of the distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns; and

estimating means to estimate head motion from identified points in the two images, the identified points being based on the set of physical face parameters; and

wherein the estimating means further comprise;

calculating means to calculate an estimation objective function to determine an initial estimate of head rotation, the estimation objective function comprising a set of terms to estimate;

(a) each of the set of physical face parameters, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image;

placing means to associate an inequality constraint on a particular face parameter of the set of physical face parameters, such that the particular face parameter is constrained within a predetermined minimum and maximum value;

converting means to convert the inequality constraint to an equality constraint using a penalty function; and

adding means to add the inequality constraint to the initial estimate.

22. A method to estimate motion between two images, the method comprising:

determining locations of a plurality of distinct features in the two images;

converting the identified locations into a set of parameters based on the symmetry of the distinct features; and

estimating motion between the two images based on the set of physical face parameters by;

(1) calculating an estimation objective function comprising a set of terms to estimate;

(a) each of the parameters, (b) a first transform corresponding to the first image, and (c) a second transform corresponding to the second image; and

(2) multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

23. A method as recited in claim 22, after converting and before estimating, further comprising adding an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value.

24. A method as recited in claim 23, wherein the predetermined minimum value and the predetermined maximum value are based on absolute values of other of the locations.

25. A method as recited in claim 23, further comprising converting the inequality constraint to an equality constraint using a penalty function.

26. A method as recited in claim 22, wherein the estimation objective function is used to determine an initial estimate of image rotation, wherein the first transform is a first pose transform corresponding to the first image, and wherein the second transform is a second pose transform corresponding to the second image, and wherein estimating further comprises:

placing an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

adding the inequality constraint to the initial estimate.

27. A method as recited in claim 26, wherein multiplying further comprises multiplying the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

28. A method as recited in claim 22, wherein determining the locations further comprises accepting input from a human user.

29. A computing device comprising:

a processor; and

a memory coupled to the processor, the memory comprising computer-program instructions executable by the processor to estimate motion between two images, the computer-program instructions comprising instructions for;

determining locations of a plurality of distinct features in the two images;

converting the identified locations into a set of parameters based on the symmetry of the distinct features; and

estimating motion between the two images based on the set of physical face parameters by;

(1) calculating an estimation objective function comprising a set of terms to estimate;

(a) each of the parameters, (b) a first transform corresponding to the first image, and (c) a second transform corresponding to the second image; and

(2) multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

30. A computing device as recited in claim 29, logically after the instructions for converting and logically before the instructions for estimating, further comprising instructions for adding an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value.

31. A computing device as recited in claim 30, wherein the predetermined minimum value and the predetermined maximum value are based on absolute values of other of the locations.

32. A computing device as recited in claim 30, further comprising instructions for converting the inequality constraint to an equality constraint using a penalty function.

33. A computing device as recited in claim 29, wherein the estimation objective function is used to determine an initial estimate of image rotation, wherein the first transform is a first pose transform corresponding to the first image, and wherein the second transform is a second pose transform corresponding to the second image, and wherein the instructions for estimating further comprise instructions for:

placing an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

adding the inequality constraint to the initial estimate.

34. A computing device as recited in claim 33, wherein the instructions for multiplying further comprise instructions for multiplying the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

35. A computing device as recited in claim 29, wherein the instructions for determining the locations further comprise instructions for accepting input from a human user.

36. A computing device comprising:

determining means to determine locations of a plurality of distinct features in the two images;

converting means to convert the identified locations into a set of parameters based on the symmetry of the distinct features; and

estimating means to estimate motion between the two images based on the set of physical face parameters by;

(1) calculating an estimation objective function comprising a set of terms to estimate;

(a) each of the parameters, (b) a first transform corresponding to the first image, and (c) a second transform corresponding to the second image; and

(2) multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

37. A computing device as recited in claim 36, logically after the converting means and logically before the estimating means, further comprising adding means to add an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value.

38. A computing device as recited in claim 37, further comprising converting means to convert the inequality constraint to an equality constraint using a penalty function.

39. A computing device as recited in claim 36, wherein the estimation objective function determines an initial estimate of image rotation, wherein the first transform is a first pose transform corresponding to the first image, and wherein the second transform is a second pose transform corresponding to the second image, and wherein the estimating means further comprise:

placing means to associate an inequality constraint on a particular parameter of the parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value;

converting means to convert the inequality constraint to an equality constraint using a penalty function; and

adding means to add the inequality constraint to the initial estimate.

40. A computing device as recited in claim 39, wherein the multiplying means multiply the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

41. A computing device as recited in claim 36, wherein the determining means further comprise accepting means to accept input from a human user.

42. A computer-readable medium to estimate head motion between two images of a face, the computer-readable medium comprising computer program instructions for:

identifying locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting the locations into set of physical face parameters based on the symmetry of the identified distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns;

determining an initial estimation of head motion between the two images using the set of physical face parameters as follows;

(1) calculating an estimation objective function comprising a number of terms to estimate each of;

(a) the set of coordinates, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image; and

(2) multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function; and

refining the initial estimation of head motion by incorporating the initial estimation into a feature matching algorithm.

43. A computer-readable medium as recited in claim 42, wherein the identified locations correspond to the eye corners, mouth corners and nose tip.

44. A computer-readable medium as recited in claim 42, wherein the identifying comprises accepting input from a human user.

45. A computer-readable medium as recited in claim 42, further comprising computer-program instructions executable by the processor for multiplying the refined head motion estimate by a confidence factor that indicates that each of the identified locations was identified with a same level of accuracy.

46. A computer-readable medium as recited in claim 42, logically after the computer-program instructions for converting and before the computer-program instructions for determining, further comprising computer-program instructions executable by the processor for adding an inequality constraint on a particular parameter of the physical face parameters, such that the parameter is constrained within a predetermined minimum and maximum value.

47. A computer-readable medium as recited in claim 46, wherein the particular parameter corresponds to the nose tip, the predetermined minimum value is zero (0), and the predetermined maximum value is a value based on absolute values of other of the locations.

48. A computer-readable medium as recited in claim 46, further comprising converting the inequality constraint to an equality constraint using a penalty function.

49. A computer-readable medium as recited in claim 42, wherein the computer-program instructions for determining further comprise instructions for:

adding an inequality constraint on a particular parameter of the physical face parameters, such that the particular parameter is constrained within a predetermined minimum and maximum value;

converting the inequality constraint to an equality constraint using a penalty function; and

adding the inequality constraint to the initial estimate.

50. A computer-readable medium as recited in claim 49, wherein in the instructions for determining further comprise instructions for multiplying the equality constraint by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function.

51. A computing device to estimate head motion between two images of a face, the computing device comprising:

identifying means to identify locations of a plurality of distinct facial features in the two images, the locations corresponding to a number of unknowns determined upon estimation of head motion, the number of unknowns being determined by a number of equations;

converting means to convert the locations into set of physical face parameters based on the symmetry of the identified distinct facial features, the set of physical face parameters reducing the number of unknowns as compared to the number of equations used to determine the unknowns;

determining means to determine an initial estimation of head motion between the two images using the set of physical face parameters as follows;

calculating an estimation objective function comprising a number of terms to estimate each of;

(a) the set of coordinates, (b) a first head pose transform corresponding to the first image, and (c) a second head pose transform corresponding to the second image; and

multiplying each term of the estimation objective function by a weighted contribution factor based on the confidence of data corresponding to the estimation objective function; and

refining the initial estimation of head motion by incorporating the initial estimation into a feature matching algorithm.