EFFICIENT FACIAL LANDMARK TRACKING USING ONLINE SHAPE REGRESSION METHOD

• US 20150169938A1
• Filed: 12/13/2013
• Published: 06/18/2015
• Est. Priority Date: 12/13/2013
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Yao, Anbang, Du, Yangzhou, Tong, Xiaofeng, Wang, Tao, Chen, Yurong, Li, Jianguo, Li, Wenlong, Zhang, Yimin, Ye, Jianbo
• Granted Patent: US 9,361,510 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 10/755 Deformable models or variational models, e.g. snakes or active contours
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

Disclosed in some examples are various modifications to the shape regression technique for use in real-time applications, and methods, systems, and machine readable mediums which utilize the resulting facial landmark tracking methods.

Claims

1. A method for detection of facial landmark points, the method comprising:

detecting a facial image in a first video frame using a face classifier;

normalizing the detected facial image by applying a first transform to the detected facial image;

detecting a set of facial landmark points of the normalized facial image using a trained shape regressor;

computing a set of final facial landmark points in the first video frame by applying a second transform to the facial landmark points, the second transform an inverse transform of the first transform.

2. The method of claim 1, comprising:

calculating a facial bounding box for a second video frame using the computed set of facial landmark points from the first video frame;

normalizing a second facial image in the second video frame by applying a third transform to the second facial image, the second facial image comprising the image of the second video frame within the facial bounding box;

detecting a second set of facial landmark points in the normalized second facial image using the trained shape regressor; and

computing a second set of final facial landmark points in the second video frame by applying a fourth transform to the facial landmark points, the fourth transform an inverse transform of the third transform.

3. The method of claim 2, comprising:

for each of two facial component groups;

compute a reference point position;

compute a sum distance of the set of facial landmark points in the second video frame to the reference point position;

measure a geometric similarity between the second set of landmark points in the second video frame and the set of landmark points in the first video frame; and

apply an importance driven filtering to produce a jitter compensated set of facial landmark points.

4. The method of claim 3, wherein applying an importance driven filter comprises weighting positions of the landmark points in the first video frame with a first weighting factor and weighting positions of the landmark points in the second video frame with a second weighting factor, and wherein the first and second weighting factors are determined based upon the measured geometric similarity.

5. The method of claim 4, comprising:

training the trained shape regressor by creating a generalized fern to partition a training dataset with at least two thresholds, wherein the generalized fern is a division of the training dataset based upon pixel intensities.

6. The method of claim 1 comprising:

animating a graphical avatar using the facial landmark points.

7. The method of claim 6, comprising:

collecting face shape information for a plurality of subjects, each respective subject in the plurality of subjects emoting a pre-defined expression;

computing a first distance metric to described the differences between the plurality of subjects; and

computing a second distance metric to describe the differences between different expressions of the plurality of subjects.

8. The method of claim 7, comprising:

capturing a neutral face of a new user;

determining a facial expression basis for the new user using the first distance metric.

9. The method of claim 8, wherein animating the graphical avatar using the facial landmark points comprises calculating a facial retargeting using a blend shape animation formula using the first and second metrics, the set of facial landmark points, and the facial expression basis for the new user as inputs.

10. A device for detection of facial landmark points, the device comprising:

one or more processors arranged to include at least;

a facial classifier module to;

detect a facial image in a first video frame; and

an online shape regression module to;

normalize the detected facial image by applying a first transform to the detected facial image;

detect a set of facial landmark points of the normalized facial image using a trained shape regressor;

compute a set of final facial landmark points in the first video frame by applying a second transform to the facial landmark points, the second transform a reverse transform of the first transform.

11. The device of claim 10, wherein the online shape regression module is to:

calculate a facial bounding box for a second video frame using the computed set of facial landmark points from the first video frame;

normalize a second facial image in the second video frame by applying a third transform to the second facial image, the second facial image comprising the image of the second video frame within the facial bounding box;

detect a second set of facial landmark points in the normalized second facial image using the trained shape regressor; and

compute a second set of final facial landmark points in the second video frame by applying a fourth transform to the facial landmark points, the fourth transform an inverse transform of the third transform.

12. The device of claim 11 wherein the online shape regression module is to pad the facial bounding box by a predetermined amount.

13. The device of claim 11, comprising:

a jitter compensation module to;

for each of two facial component groups;

compute a reference point position;

compute a sum distance of the set of facial landmark points in the second video frame to the reference point position;

measure a geometric similarity between the second set of landmark points in the second video frame and the set of landmark points in the first video frame; and

apply an importance driven filtering to produce a jitter compensated set of facial landmark points.

14. The device of claim 13, wherein the jitter compensation module is to apply an importance driven filter by being configured to at least weight positions of the landmark points in the first video frame with a first weighting factor and weight positions of the landmark points in the second video frame with a second weighting factor, and wherein the first and second weighting factors are determined based upon the measured geometric similarity.

15. The device of claim 14, comprising:

a shape regression training module to;

train the trained shape regressor by creating a generalized fern to partition a training dataset with at least two thresholds, wherein the fern is a division of the training dataset based upon pixel intensities.

16. The device of claim 10 comprising:

a facial expression retargeting module to animate a graphical avatar using the facial landmark points.

17. A machine readable medium for detection of facial landmark points, the machine readable medium storing instructions, which when performed by a machine, cause the machine to perform operations comprising:

detecting a facial image in a first video frame using a face classifier;

normalizing the detected facial image by applying a first transform to the detected facial image;

detecting a set of facial landmark points of the normalized facial image using a trained shape regressor; and

computing a set of final facial landmark points in the first video frame by applying a second transform to the facial landmark points, the second transform an inverse transform of the first transform.

18. The machine readable medium of claim 17, wherein the instructions include instructions, which when performed by the machine, cause the machine to perform the operations comprising:

calculating a facial bounding box for a second video frame using the computed set of facial landmark points from the first video frame;

normalizing a second facial image in the second video frame by applying a third transform to the second facial image, the second facial image comprising the image of the second video frame within the facial bounding box;

detecting a second set of facial landmark points in the normalized second facial image using the trained shape regressor; and

computing a second set of final facial landmark points in the second video frame by applying a fourth transform to the facial landmark points, the fourth transform an inverse transform of the third transform.

19. The machine readable medium of claim 18 wherein the instructions include instructions, which when performed by the machine, cause the machine to perform the operations comprising:

padding the facial bounding box by a predetermined amount.

20. The machine readable medium of claim 18 wherein the instructions include instructions, which when performed by the machine, cause the machine to perform the operations comprising:

for each of two facial component groups;

compute a reference point position;

compute a sum distance of the set of facial landmark points in the second video frame to the reference point position;

measure a geometric similarity between the second set of landmark points in the second video frame and the set of landmark points in the first video frame; and

apply an importance driven filtering to produce a jitter compensated set of facial landmark points.

21. The machine readable medium of claim 20, wherein the instructions for applying an importance driven filter comprises instructions, which when performed by the machine, cause the machine to perform operations comprising:

weighting positions of the landmark points in the first video frame with a first weighting factor and weighting positions of the landmark points in the second video frame with a second weighting factor, and wherein the first and second weighting factors are determined based upon the measured geometric similarity.

22. The machine readable medium of claim 21, wherein the instructions include instructions, which when performed by the machine, cause the machine to perform the operations comprising:

training the trained shape regressor by creating a generalized fern to partition a training dataset with at least two thresholds, wherein the fern is a division of the training dataset based upon pixel intensities.

23. The machine readable medium of claim 17 wherein the instructions include instructions, which when performed by the machine, cause the machine to perform the operations comprising:

animating a graphical avatar using the facial landmark points.

24. The machine readable medium of claim 23, wherein the instructions include instructions, which when performed by the machine, cause the machine to perform the operations comprising:

collecting face shape information for a plurality of subjects, each respective subject in the plurality of subjects emoting a pre-defined expression;

computing a first distance metric to described the differences between the plurality of subjects;

computing a second distance metric to describe the differences between different expressions of the plurality of subjects;

capturing a neutral face of a new user; and

determining a facial expression basis for the new user using the first distance metric; and

wherein the instructions for animating the graphical avatar using the facial landmark points comprise instructions, which include instructions, which when performed by the machine, cause the machine to perform the operations of calculating a facial retargeting using a blend shape animation formula using the first and second metrics, the set of facial landmark points, and the facial expression basis for the new user as inputs.

25. The machine readable medium of claim 17, wherein the first transform is a scaling transform.