FACE RECOGNITION METHOD AND APPARATUS

• US 20160379041A1
• Filed: 06/21/2016
• Published: 12/29/2016
• Est. Priority Date: 06/24/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: RHEE, Seon Min, KIM, Jungbae, YOO, Byungin, HAN, Jaejoon, HAN, Seungju
• Granted Patent: US 10,331,941 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2021 Shape modification
• CPC Class Codes: G06V 10/32 Normalisation of the pattern dimensions
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

Face recognition of a face, to determine whether the face correlates with an enrolled face, may include generating a personalized three-dimensional (3D) face model based on a two-dimensional (2D) input image of the face, acquiring 3D shape information and a normalized 2D input image of the face based on the personalized 3D face model, generating feature information based on the 3D shape information and pixel color values of the normalized 2D input image, and comparing the feature information with feature information associated with the enrolled face. The feature information may include first and second feature information generated based on applying first and second deep neural network models to the pixel color values of the normalized 2D input image and the 3D shape information, respectively. The personalized 3D face model may be generated based on transforming a generic 3D face model based on landmarks detected in the 2D input image.

Claims

1. A face recognition method, comprising:

generating a personalized three-dimensional (3D) face model based on a two-dimensional (2D) input image of a user face;

generating 3D shape information based on the personalized 3D face model, the 3D shape information including one or more sets of 3D surface coordinate values;

generating a normalized 2D input image of the user face based on the personalized 3D face model;

generating feature information associated with the user face, based on the 3D shape information and pixel color values of the normalized 2D input image; and

performing face recognition based on the feature information.

2. The face recognition method of claim 1, wherein,

the generating the feature information associated with the user face includes,

generating first feature information based on applying a first deep neural network model to the pixel color values of the normalized 2D input image, and

generating second feature information based on applying a second deep neural network model to the 3D shape information of the personalized 3D face model.

3. The face recognition method of claim 1, wherein the generating of the 3D shape information includes,

generating a depth image of the user face based on the personalized 3D face model, the depth image including a 3D shape associated with the user face;

adjusting the 3D shape based on the normalized 2D input image of the user face; and

generating the 3D shape information from the adjusted 3D shape.

4. The face recognition method of claim 3, wherein the adjusting the 3D shape includes,

identifying one or more meshes associated with a plurality of feature points in the normalized 2D input image;

identifying one or more meshes associated with a plurality of feature points in the depth image; and

adjusting the 3D shape based on a corresponding relationship between the one or more meshes associated with the normalized 2D input image and the one or more meshes associated with the depth image.

5. The face recognition method of claim 1, wherein the generating of the 3D shape information includes,

detecting one or more landmarks of the user face in the 2D input image, the one or more landmarks including one or more feature points indicating one or more boundaries or interfaces between different surfaces of the user face;

and

transforming a generic 3D face model to the personalized 3D face model based on the detected one or more landmarks.

6. The face recognition method of claim 1, wherein the generating of the 3D shape information includes,

generating one of a depth image of the user face and a normal image of the user face based on the personalized 3D face model;

and

generating the 3D shape information from at least one of the depth image and the normal image.

7. The face recognition method of claim 2, wherein the generating the feature information associated with the user face includes,

generating the first feature information based on applying the first deep neural network model to pixel depth values of a depth image and the pixel color values of the normalized 2D input image, the depth image being generated based on the personalized 3D face model.

8. The face recognition method of claim 2, wherein the generating the feature information associated with the user face includes,

generating the first feature information based on applying the first deep neural network model to surface normal information of a normal image of the user face, the normal image being generated based on the personalized 3D face model and pixel color values of the 2D input image.

9. The face recognition method of claim 2, wherein the second deep neural network model is an auto-encoder structured neural network model.

10. The face recognition method of claim 1, wherein the generating the normalized 2D input image includes,

generating pose information associated with a frontal face, based on the personalized 3D face model;

and

normalizing the 2D input image based on the pose information.

11. The face recognition method of claim 1, wherein the generating the normalized 2D input image includes,

detecting one or more landmarks associated with the user face based on the 2D input image of the user face;

and

normalizing the 2D input image based on the one or more detected landmarks.

12. The face recognition method of claim 2, wherein the performing of the face recognition includes,

generating final feature information associated with the 2D input image based on the first feature information and the second feature information;

and

generating a face recognition result based on comparing final feature information associated with the 2D input image to feature information associated with a stored 2D image.

13. The face recognition method of claim 1, wherein the performing of the face recognition includes determining whether the user face included in the 2D input image correlates with a stored 2D image of an enrolled user face, based on determining whether the feature information associated with the user face correlates with feature information associated with the enrolled user face.

14. The face recognition method of claim 13, wherein the performing of the face recognition further includes generating an unlocking command signal to enable user access to at least a portion of a terminal device, based on determining that the user face included in the 2D input image correlates with the stored 2D image of the enrolled user face.

15. A non-transitory computer-readable storage medium storing computer-readable instructions, the computer-readable instructions being executable by a processor to cause the processor to perform the method of claim 1.

16. An apparatus comprising:

a memory storing computer-readable instructions; and

a processor configured to execute the computer-readable instructions to,

detect one or more landmarks of a user face based on processing a two-dimensional (2D) image of the user face;

generate a personalized three-dimensional (3D) face model based on the detected one or more landmarks;

generate 3D shape information associated with the user face based on processing the personalized 3D face model;

generate first feature information associated with the user face, based on applying a first deep neural network model to pixel color values of the 2D image of the user face; and

generate second feature information associated with the user face, based on applying a second deep neural network model to the 3D shape information, wherein the first deep neural network model differs from the second deep neural network model.

17. The apparatus of claim 16, wherein the generating the first feature information includes applying the first deep neural network model to the pixel color values of the 2D image to generate a feature associated with an appearance of the user face.

18. The apparatus of claim 16, wherein the generating the second feature information includes applying the second deep neural network model to the 3D shape information to generate a feature associated with a shape of the user face.

19. The apparatus of claim 16, wherein the processor is configured to execute computer-readable instructions to,

normalize the image based on at least one of the one or more detected landmarks and the personalized 3D face model.

20. A face recognition apparatus comprising:

a memory storing computer-readable instructions; and

a processor configured to execute the computer-readable instructions to, generate a personalized three-dimensional (3D) face model based on a two-dimensional (2D) input image of a user face;

generate 3D shape information based on the personalized 3D face model, the 3D shape information including one or more sets of 3D surface coordinate values;

generate a normalized 2D input image of the user face based on the personalized 3D face model;

generate feature information associated with the user face, based on the 3D shape information and pixel color values of the normalized 2D input image;

perform face recognition based on the feature information.