3D FACE MODEL RECONSTRUCTION APPARATUS AND METHOD

• US 20160275721A1
• Filed: 06/20/2014
• Published: 09/22/2016
• Est. Priority Date: 06/20/2014
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation, Minje Park, Myung-He Ju, Tae- Hoon Kim, Yangzhou Du, Yeongjae Cheon
• Inventors: RYU, Wooju, JU, Myung-Ho, PARK, Minje, DUCHENNE, Olivier, CHEON, Yeongjae, KIM, Tae-Hoon, SHEN, Xiaolu, DU, Yangzhou
• Granted Patent: US 9,679,412 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 17/205 Re-meshing
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

Apparatuses, methods and storage medium associated with 3D face model reconstruction are disclosed herein. In embodiments, an apparatus may include a facial landmark detector, a model fitter and a model tracker. The facial landmark detector may be configured to detect a plurality of landmarks of a face and their locations within each of a plurality of image frames. The model fitter may be configured to generate a 3D model of the face from a 3D model of a neutral face, in view of detected landmarks of the face and their locations within a first one of the plurality of image frames. The model tracker may be configured to maintain the 3D model to track the face in subsequent image frames, successively updating the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames. In embodiments, the facial landmark detector may include a face detector, an initial facial landmark detector, and one or more facial landmark detection linear regressors. Other embodiments may be described and/or claimed.

Claims

1. An apparatus equipped to construct a three-dimensional, 3D, face model, comprising:

a facial landmark detector to detect a plurality of landmarks of a face, and their locations, within each of a plurality of image frames;

a model fitter coupled to the facial landmark detector to generate a 3D model of the face from a 3D model of a neutral face, in view of detected landmarks of the face and their locations within a first one of the plurality of image frames; and

a model tracker coupled to the facial landmark detector and the model fitter to adjust the 3D model to track the face in subsequent image frames, successively adjusting the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.

2. The apparatus of claim 1, wherein the facial landmark detector comprises a face detector to detect a face within an image frame;

an initial facial landmark detector coupled with the face detector to detect one or more landmarks, including an inner eye corner, and their locations within a detected face;

one or more facial landmark detection linear regressors coupled with the face detector and the initial facial landmark detector to iteratively refine the facial landmark locations of the detected face with increased precision.

3. The apparatus of claim 2, wherein the facial landmark detector further comprises a validator coupled with the face detector and the initial facial landmark detector to reconstruct a face from the facial landmarks, and an image frame, and provide the reconstructed image frame to the face and initial facial landmark detectors to detect whether a face and one or more eyes are present within the reconstructed image frame.

4. The apparatus of claim 2, wherein the one or more face landmark detection linear regressors comprise a first face landmark detection linear regressor to refine facial landmark locations to a first precision, and a second facial landmark detection linear regressor to refine the facial landmark locations from the first precision to a second precision, wherein the second precision is more precise than the first precision.

5. The apparatus of claim 1, wherein the model fitter to estimate an initial 3D mesh and head pose, in view of the detected landmarks of the face and their locations within the first image frame.

6. The apparatus of claim 5, wherein the model fitter to further modify the initial 3D mesh and head pose to reduce a first amount of differences between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected by the facial landmark detector.

7. The apparatus of claim 5, wherein the model fitter to further modify the initial 3D mesh and head pose to reduce a second amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected by the facial landmark detector.

8. The apparatus of claim 5, wherein the model fitter to further modify the initial 3D mesh and head pose to reduce a third amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected by the facial landmark detector.

9. The apparatus of claim 5, wherein the model fitter to further modify the initial 3D mesh and head pose to reduce a sum of a first, a second and a third amount of differences, wherein the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected, the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected, and the third amount of differences is an amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.

10. The apparatus of claim 1, wherein the model tracker to fix inter-person shape variations, and consider only intra-person expression variation, when successively updating the 3D model in view of detected landmarks of the face and their locations within successive ones of the plurality of image frames.

11. The apparatus of claim 1, further comprising a validator coupled with the model fitter and the model tracker to monitor the adjustment of the 3D model to track the face to detect for excessive drifting, and to cause model fitter to re-generate another 3D model of the face when excessive drifting is detected.

12. The apparatus of claim 11, wherein the validator to detect for excessive drifting through detection of whether inaccuracies of the 3D model successively maintained by the model tracker has grown to exceed a threshold.

13. (canceled)

14. A method for constructing a three-dimensional, 3D, face model, comprising:

detecting, by a computing device, a plurality of landmarks of a face, and their locations, within each of a plurality of image frames;

generating, by the computing device, in view of detected landmarks of the face and their locations within a first one of the plurality of image frames; and

adjusting, by the computing device, the 3D model to track the face in subsequent image frames, successively adjusting the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.

15. The method of claim 14, wherein detecting a plurality of landmarks of a face comprises:

detecting a face within an image frame;

detecting one or more landmarks, including an inner eye corner, and their locations within a detected face; and

iteratively refining the facial landmark locations of the detected face with increased precision.

24. The method of claim 14, further comprising monitoring the adjustment of the 3D model to track the face to detect for excessive drifting, and re-generating another 3D model of the face when excessive drifting is detected.

16-23. -23. (canceled)

25-34. -34. (canceled)

35. One or more computer-readable medium having instructions to cause a computing device, in response to execution of the instructions by the computing device, to:

detect a plurality of landmarks of a face, and their locations, within each of a plurality of image frames;

generate in view of detected landmarks of the face and their locations within a first one of the plurality of image frames; and

adjust the 3D model to track the face in subsequent image frames, including successively adjust the 3D model in view of detected landmarks of the face and their locations within each of successive ones of the plurality of image frames.

36. The computer-readable medium of claim 35, wherein to detect a plurality of landmarks of a face comprises:

detect a face within an image frame;

detect one or more landmarks, including an inner eye corner, and their locations within a detected face; and

iteratively refine the facial landmark locations of the detected face with increased precision.

37. The computer-readable medium of claim 36, wherein to detect a plurality of landmarks of a face further comprises reconstruct a face from the facial landmarks, and an image frame, and detect whether a face and one or more eyes are present within the reconstructed image frame.

38. The computer-readable medium of claim 36, wherein iteratively refine comprises first refine facial landmark locations to a first precision, and second refine the facial landmark locations from the first precision to a second precision, wherein the second precision is more precise than the first precision.

39. The computer-readable medium of claim 35, wherein to generate comprises estimate an initial 3D mesh and head pose, in view of the detected landmarks of the face and their locations within the first image frame.

40. The computer-readable medium of claim 39, wherein to generate further comprises modify the initial 3D mesh and head pose to reduce a first amount of differences between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected, a second amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected, or a third amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.

41. The computer-readable medium of claim 39, wherein to generate further comprises modify the initial 3D mesh and head pose to reduce a sum of a first, a second and a third amount of differences, wherein the first amount of differences is an amount of difference between locations of the landmarks as depicted by the 3D mesh and head pose and locations of the landmarks as detected, the second amount of differences is an amount of differences between locations of the landmarks as depicted by an ellipsoidal model of the face and locations of the landmarks as detected, and the third amount of differences is an amount of differences between locations of the landmarks as depicted by a deformable mesh model of the face and locations of the landmarks as detected.

42. The computer-readable medium of claim 35, wherein to adjust comprises fix inter-person shape variations, and consider only intra-person expression variation, when successively update the 3D model in view of detected landmarks of the face and their locations within successive ones of the plurality of image frames.

43. The computer-readable medium of claim 35, wherein the computing device is further caused to monitor the adjustment of the 3D model to track the face to detect for excessive drifting, and re-generate another 3D model of the face when excessive drifting is detected.

44. The computer-readable medium of claim 43, wherein to monitor comprises detect whether inaccuracies of the 3D model successively maintained by the model tracker has grown to exceed a threshold.