Three-dimensional face capturing apparatus and method and computer-readable medium thereof

• US 8,803,950 B2
• Filed: 08/24/2010
• Issued: 08/12/2014
• Est. Priority Date: 08/24/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ren, Haibing, Huang, Xiangsheng, Lim, Hwa Sup
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Anderson, Matthew J
• Application Number:
  US12/862,293
• Publication Number:
  US 20110043610A1
• Time in Patent Office: 1,449 Days
• Field of Search: 348 42- 50, 382/154, 382/118, 345/419, 345/420
• US Class Current: 348/46
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body

Abstract

Disclosed is a 3D face capturing apparatus, method and computer-readable medium. As an example, the 3D face capturing method includes obtaining a face color image, obtaining a face depth image, aligning, by a computer, the face color image and the face depth image, obtaining, by the computer, a 3D face model by 2D modeling of the face color image and covering a modeled 2D face area on an image output by an image alignment module, removing by the computer, depth noise of the 3D face model, and obtaining, by the computer, an accurate 3D face model by aligning the 3D face model and a 3D face template, and removing residual noise based on a registration between the 3D face model and the 3D face template.

Claims

1. An apparatus capturing a three-dimensional (3D) face, the apparatus comprising:

a color image obtaining unit to obtain a face color image;

a depth image obtaining unit to obtain a face depth image;

an image alignment module to align the obtained face color image and the obtained face depth image;

a 3D face model generating module to perform two-dimensional (2D) modeling of the obtained face color image, and to perform covering of a modeled 2D face area on an image output by the image alignment module to obtain a 3D face model;

a first denoising module to remove depth noise of the 3D face model; and

a second denoising module to align the 3D face model and a 3D face template, and to remove residual noise based on a registration between the 3D face model and the 3D face template to obtain an accurate 3D face model,

wherein the 3D face model generating module performs the 2D modeling using an ellipse face color image, and

wherein the first denoising module calculates a scale-independent variance (V) in a depth direction, and compares V with a first threshold to determine whether noise exists in the 3D face model.

2. The apparatus of claim 1, wherein a method of calculating V satisfies Equation 10:

3. The apparatus of claim 1, wherein the first threshold is selected based on experimentation.

4. The apparatus of claim 1, wherein, when the determining determines that the noise exists in the 3D face model, the first denoising module calculates a depth direction average value of vertices ( z), compares a second threshold (T) with a difference between a depth of a vertex and z, and removes the vertex as the noise when the difference is greater than T.

5. The apparatus of claim 4, wherein z is calculated by substitution based on Equation 11:

6. The apparatus of claim 4, wherein T satisfies Equation 12:

7. The apparatus of claim 1, wherein the second denoising module performs:

calculating a length factor,

unification with respect to the input 3D face template based on the calculated length factor;

a vertex registration with respect to the 3D face model and the 3D face template based on features;

calculating a mobility coefficient and a rotation coefficient with respect to a vertex of the vertex registration;

determining whether the mobility coefficient and the rotation coefficient converge;

updating the length factor when the mobility coefficient and the rotation coefficient do not converge;

updating the 3D face model based on the updated length factor; and

repeating the calculating the length factor, unification, vertex registration, calculating the mobility and the rotation coefficient, determining, updating the length factor and updating the 3D face model until the mobility coefficient and the rotation coefficient converge.

8. The apparatus of claim 7, wherein the features include 3D coordinates of a vertex, color information, a normal direction, and information associated with an area adjacent to the vertex.

9. The apparatus of claim 1, wherein the color image obtaining module is a CCD camera.

10. The apparatus of claim 1, wherein the depth image obtaining module is a depth camera.

11. A method of capturing a 3D face, the method comprising:

obtaining a face color image;

obtaining a face depth image;

aligning, by a computer, the obtained face color image and the obtained face depth image;

obtaining, by the computer, a 3D face model by 2D modeling of the obtained face color image and covering a modeled 2D face area on an image output by an image alignment module;

removing, by the computer, depth noise of the 3D face model;

and obtaining, by the computer, an accurate 3D face model by aligning the 3D face model and a 3D face template, and removing residual noise based on a registration between the 3D face model and the 3D face template,

wherein the 2D modeling performs the 2D modeling with respect to the face color image using an ellipse, and

wherein a scale-independent variance (V) is calculated in a depth direction, and compares V with a first threshold to determine whether noise exists in the 3D face model.

12. The method of claim 11, wherein a method of calculating V satisfies Equation 13:

13. The method of claim 11, wherein the first threshold is selected based on experimentation.

14. The method of claim 11, wherein the removing comprises:

calculating a depth direction average value of vertices ( z) when the noise exists in the 3D face model;

comparing a second threshold (T) with a difference between a depth of a vertex and z; and

removing the vertex as the noise when the difference is greater than T.

15. The method of claim 14, wherein the z is calculated by substitution based on Equation 14:

16. The method of claim 14, wherein T satisfies Equation 15:

17. The method of claim 11, wherein the aligning comprises:

calculating a length factor;

performing unification with respect to the input 3D face template based on the calculated length factor;

performing a vertex registration with respect to the 3D face model and the 3D face template based on features;

determining whether a mobility coefficient and a rotation coefficient converge by calculating the mobility coefficient and the rotation coefficient with respect to a vertex of the vertex registration;

updating the length factor when the mobility coefficient and the rotation coefficient do not converge;

updating the 3D face model based on the updated length factor; and

repeating the calculating the length factor, unification, vertex registration, calculating the mobility and the rotation coefficient, determining, updating the length factor and updating the 3D face model until the mobility coefficient and the rotation coefficient converge.

18. The method of claim 17, wherein the features include 3D coordinates of a vertex, color information, a normal direction, and information associated with an area adjacent to the vertex.

19. The method of claim 17, wherein, when a vertex that is not registered exists in the 3D face model, the performing of the vertex registration comprises removing the vertex as noise.

20. At least one non-transitory computer readable recording medium comprising computer readable instructions that control at least one processor to implement a method, comprising:

obtaining a face color image;

obtaining a face depth image;

aligning the obtained face color image and the obtained face depth image;

obtaining a 3D face model by 2D modeling of the obtained face color image and covering a modeled 2D face area on an image output by an image alignment module;

removing depth noise of the 3D face model; and

obtaining an accurate 3D face model by aligning the 3D face model and a 3D face template, and removing residual noise based on a registration between the 3D face model and the 3D face template,

wherein the 2D modeling is performed using an ellipse face color image, and

wherein a scale-independent variance (V) is calculated in a depth direction, and compares V with a first threshold to determine whether noise exists in the 3D face model.