METHOD AND APPARATUS FOR ADJUSTING FACE POSE

• US 20160210500A1
• Filed: 01/15/2016
• Published: 07/21/2016
• Est. Priority Date: 01/15/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: FENG, Hao, CHOI, Changkyu, WANG, Biao, FENG, Xuetao, ZHANG, Hongwei, GONG, Lujin, HAN, Jae-Joon, SUH, Sungjoo, HWANG, Wonjun
• Granted Patent: US 10,134,177 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06V 10/54 relating to texture
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

A method and an apparatus for adjusting a pose in a face image are provided. The method of adjusting a pose in a face image involves detecting two-dimensional (2D) landmarks from a 2D face image, positioning three-dimensional (3D) landmarks in a 3D face model by determining an initial pose of the 3D face model based on the 2D landmarks, updating the 3D landmarks by iteratively adjusting a pose and a shape of the 3D face model, and adjusting a pose in the 2D face image based on the updated 3D landmarks.

Claims

1. A method of adjusting a pose in a face image, comprising:

detecting two-dimensional (2D) landmarks from a 2D face image;

positioning three-dimensional (3D) landmarks in a 3D face model by determining an initial pose of the 3D face model based on the 2D landmarks;

updating the 3D landmarks by iteratively adjusting a pose and a shape of the 3D face model; and

adjusting a pose in the 2D face image based on the updated 3D landmarks.

2. The method of claim 1, wherein the positioning of the 3D landmarks comprises determining an initial geometric parameter representing the initial pose of the 3D face model based on the 2D landmarks.

3. The method of claim 2, wherein the initial geometric parameter comprises initial pose information on the 3D face model, and the initial pose information is represented by a pose information vector comprising roll information, pitch information, and yaw information on the 3D face model.

4. The method of claim 2, wherein the determining of the initial geometric parameter comprises:

determining an initial weight for the 2D landmarks based on the 2D landmarks; and

determining the initial geometric parameter that minimizes a first energy function based on the initial weight, the 2D landmarks, and the 3D landmarks.

5. The method of claim 4, wherein the determining of the initial weight comprises:

measuring deviations between the 2D landmarks and an actually detected landmark of a 2D face sample image stored in a face image database; and

determining the initial weight for the 2D landmarks based on the measured deviations.

6. The method of claim 5, wherein the determining of the initial weight for the 2D landmarks based on the measured deviations comprises:

arranging the 2D landmarks based on the measured deviation;

selecting some 2D landmarks having measured deviations smaller than a preset threshold among the arranged 2D landmarks; and

assigning the initial weight to the selected 2D landmarks in inverse proportion to measured deviations.

7. The method of claim 5, wherein the face image database comprises at least one of a plurality of 2D face sample images, actually detected landmarks from the 2D face sample images, and actually measured geometric parameters from the 2D face sample images.

8. The method of claim 2, wherein the updating of the 3D landmarks comprises:

updating the 3D landmarks based on the initial geometric parameter;

determining a weight for the updated 3D landmarks; and

determining, based on the weight, a geometric parameter and a shape parameter of the 3D face model that minimize a second energy function.

9. The method of claim 8, wherein the updating of the 3D landmarks based on the initial geometric parameter comprises:

adjusting the pose of the 3D face model based on the initial geometric parameter;

acquiring a 2D texture image by two-dimensionally projecting the 3D face model with the adjusted pose;

detecting new 2D landmarks from the 2D texture image;

retrieving 3D landmarks corresponding to the new 2D landmarks; and

updating the 3D landmarks using the retrieved 3D landmarks.

10. The method of claim 9, wherein the determining of the weight for the updated 3D landmarks comprises:

acquiring current pose information on the 3D face model from the initial geometric parameter;

retrieving pose information substantially corresponding to the current pose information on the 3D face model from a previously stored face image database;

retrieving a 2D face sample image corresponding to the retrieved pose information from the face image database; and

determining a weight for 2D landmarks corresponding to the retrieved 2D face sample image based on second deviations between the 2D landmarks and the new 2D landmarks.

11. The method of claim 10, wherein the determining of the weight for the 2D landmarks comprises:

calculating an average of the second deviations; and

determining the weight for the 2D landmarks in inverse proportion to the average of the second deviations.

12. The method of claim 10, wherein the pose information substantially corresponding to the current pose information comprises pose information with a deviation from the current pose information being within a preset threshold range.

13. The method of claim 8, wherein the updating of the 3D landmarks comprises iteratively adjusting the pose and shape of the 3D face model based on whether a deviation between a geometric parameter of the 3D face model acquired in (i−

2)th iteration and a geometric parameter acquired in (i−

1)th iteration is smaller than a preset threshold or whether iteration is performed a preset number of times.

14. The method of claim 13, wherein the adjusting of the pose in the 2D face image comprises:

reconstructing the 3D face model based on a final geometric parameter and a final shape parameter calculated in final iteration; and

adjusting the pose in the 2D face image by two-dimensionally projecting the reconstructed 3D face model.

15. The method of claim 14, further comprising mapping a background preserving texture to the 2D face image with the adjusted pose.

16. The method of claim 15, wherein the mapping of the background preserving texture comprises determining a 3D texture model by imparting a color to the reconstructed 3D face model based on a pixel color of the 2D face image;

and determining a pixel color for a background region other than a face region in a face image obtained by two-dimensionally projecting the 3D texture model.

17. The method of claim 16, wherein the determining of the 3D texture model comprises:

fitting vertexes of the reconstructed 3D face model to the 2D face image based on the final geometric parameter and the final shape parameter;

assigning a texture of the fitted 2D face image to the vertexes of the reconstructed 3D face model;

determining an occluded vertex in the 3D face model based on the final geometric parameter and the final shape parameter; and

assigning a texture to the occluded vertex.

18. The method of claim 17, wherein the assigning of the texture to the occluded vertex comprises:

determining whether a symmetric point of the occluded vertex is occluded in the 3D face model; and

assigning an average texture value of the 3D texture model to the occluded vertex in response to the symmetric point being determined to be occluded, and assigning a texture of the symmetric point to the occluded vertex in response to the symmetric point being determined not to be occluded.

19. The method of claim 16, wherein the determining of the pixel color for the background region comprises:

positioning additional landmarks for the background region in the two-dimensionally projected face image;

retrieving a corresponding point to the additional landmarks from the 3D texture model; and

acquiring a color value for the corresponding point to the additional landmarks by performing piece-wise affine transformation.

20. A non-transitory computer readable medium storing therein instructions that cause a computer with hardware to perform the method of claim 1.

22. The apparatus of claim 1, further comprising:

a camera configured to capture the 2D face image; and

a receiver configured to receive the 2D face image and provide the received 2D face image to the processor.

21. An apparatus for adjusting a face pose in an image, the apparatus comprising:

a processor configured to detect two-dimensional (2D) landmarks on a 2D face image, position three-dimensional (3D) landmarks in a 3D face model stored in a memory based on the detected 2D landmarks, iteratively adjust locations of the 3D landmarks on the 3D face model, and adjust a face pose in the 2D face image based on the locations of the 3D landmarks.

23. An apparatus for recognizing a person based on a 2D image, the apparatus comprising:

the apparatus for adjusting a face pose in an image of claim 21,

wherein the processor is further configured to compare the adjusted face pose in the 2D face image with a plurality of face images stored in a database to recognize a person whose face is captured in the 2D face image.