Method for driving virtual facial expressions by automatically detecting facial expressions of a face image

• US 20080037836A1
• Filed: 08/09/2006
• Published: 02/14/2008
• Est. Priority Date: 08/09/2006
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Chen, Ji, Wang, Jin, Wang, Lin
• Granted Patent: US 7,751,599 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

A method for driving virtual facial expressions by automatically detecting facial expressions of a face image is applied to a digital image capturing device. The method includes the steps of detecting a face image captured by the image capturing device and images of a plurality of facial features with different facial expressions to obtain a key point position of each facial feature on the face image; mapping the key point positions to a virtual face as the key point positions of corresponding facial features on the virtual face; dynamically tracking the key point of each facial feature on the face image; estimating the key point positions of each facial feature of the current face image according to the key point positions of each facial feature on a previous face image; and correcting the key point positions of the corresponding facial features on the virtual face.

Claims

1. A method for driving virtual facial expressions by automatically detecting facial expressions of a face image, which is applied to a digital image capturing device, comprising:

a face detection block, for automatically detecting positions of a face, eyes, a nose, and a mouth in a target image captured by said digital image capturing device, and converting said detected positions into a standard image for a virtual front face, and calculating initial positions of all related key points of eyes, nose, and mouth of said target image on said standard image;

a key point detection block, for calculating exact positions of said key points of said target image on said standard image, and inversely aligning said exact positions of all key points of said standard image onto said target image, and labeling said exact positions as exact positions of said key points on said target image; and

a key point tracking block, for automatically tracking the positions of all key points of other target image captured by said digital image capturing device later, and correcting said exact position of each key point on said standard image.

2. The method of claim 1, wherein said standard image is a greyscale figure having a fixed size and using a front face as a center without an inclination.

3. The method of claim 2, wherein said face detection block further comprises:

obtaining an average position of each key point of said standard image on said face image sample by training and correcting a large number of front face image samples; and

performing a fitting or regression for the positions and average positions of eyes, nose and mouth on said target image to calculate initial positions of said key points on said standard image.

4. The method of claim 3, wherein said key point detection block further comprises:

using a Gabor wavelet algorithm to sample a series of Gabor Jet of each key point in said face image samples to form a Gabor Jet Bunch; and

using a point in a small neighborhood of an initial position of each key point as a selecting point, when the exact positions of all key points on said standard image are calculated, and then comparing the Gabor Jet at the initial position of each key point with each Gabor Jet in said Gabor Jet Bunch, and selecting a selecting point having the highest similarity with a certain Gabor Jet in said Gabor Jet Bunch as an exact position of the key point of said target image corresponding to said standard image.

5. The method of claim 4, wherein said key point tracking block further comprises:

obtaining a motion parameter (dx, dy) between corresponding key points of said two target images by an optical flow technique;

calculating an error ε

(dx, dy) of said motion parameter (dx, dy) between correspnding key points of said two target images according to the following formula;

ε

(d)=ε

(dx,dy)=Σ

Σ

(I(x,y)−

J(x+dx,y+dy))².

to find dx, dy that minimize said error ε

(dx, dy) and obtain an estimated position of a corresponding key point of said current target image based on the position of a specific point on said previous target image;

performing a fitting or a regression by an estimated position of said key point on said previous target image and said key point positions on said average face to calculate estimated positions of said standard image by all key points of said current target image.

6. The method of claim 5, wherein said key point tracking block further comprises:

using a Gabor wavelet algorithm to sample a series of Gabor Jets in each key point of said face image sample to form a Gabor Jet Bunch;

using a point in a small neighborhood of an estimated position of each key point as a selecting point, when the exact positions of all key points on said standard image for said current target image are calculated, and then comparing the Gabor Jet at the estimated position of all key points on said standard image for said current target image with each Gabor Jet in said Gabor Jet Bunch, and selecting a selecting point having the highest similarity with a certain Gabor Jet in said Gabor Jet Bunch as an exact position of the key point of said target image corresponding to said standard image.

7. The method of claim 6, further comprising an accuracy measurement module, and said accuracy measurement module further comprising:

comparing the similarities of optical flows between a current target image and a previous target image, and examining an average similarity of a Gabor characteristic of all key points on said target image to instantly measure a currently tracked accuracy; and

determining whether or not the accuracy of said key points matches with a predetermined standard;

if yes, then the tracking is determined as successful, or else the tracking is determined as failed, and resetting said face detection block, and detecting and calibrating a face and its key points on said face again.

8. The methods of claims 1, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes, then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

9. The methods of claims 2, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes, then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

10. The methods of claims 3, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes, then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

11. The methods of claims 4, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes, then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

12. The methods of claims 5, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes, then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

13. The methods of claims 6, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes,.then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

14. The methods of claims 7, further comprising a key point distribution adjustment module, and said key point distribution adjustment module comprising:

training a large number of front face samples to obtain a key point distribution model, and said key point distribution model being used to represent a distribution statistic of each key point on said front face and a distribution of the position of said key points on a standard image; and

determining whether or not the positions of all key points on said standard image deviate too much from an average position calculated from the statistics of said key point distribution model;

if yes, then said deviated position are pulled back to a more accurate position close to said average position calculated from the statistics of said key point distribution model.

15. The method of claim 8, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.

16. The method of claim 9, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.

17. The method of claim 10, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.

18. The method of claim 11, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.

19. The method of claim 12, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.

20. The method of claim 13, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.

21. The method of claim 14, wherein said digital image capturing device defines 28 key points for said captured face image, and both eyebrows define four key points which are situated at the highest position of the upper edge of left and right eyebrows, tips of the internal side of the upper edge of left and right eyebrows;

both eyes define eight key points which are points at the corners of eyes and the middle point of eyelids;

a nose defines four key points which are points at both sides of the wing of said nose, a point at the bridge of said nose and a point at the tip of said nose; and

a mouth defines 12 key points which are points disposed equidistantly along the external edge of a lip.