Method and apparatus for reconstructing 3D face with stereo camera

• US 10,043,278 B2
• Filed: 01/30/2015
• Issued: 08/07/2018
• Est. Priority Date: 02/10/2014
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Kap Kee, Yoon, Seung Uk, Hwang, Bon Woo, Lim, Seong Jae, Jun, Hye Ryeong, Choi, Jin Sung, Koo, Bon Ki
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Dang, Philip
• Application Number:
  US14/609,887
• Publication Number:
  US 20150228081A1
• Time in Patent Office: 1,285 Days
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Provided is a method and apparatus for reconstructing a three-dimensional (3D) face based on a stereo camera, the method including: acquiring n images of a target by controlling a plurality of stereo cameras in response to an image acquirement request, wherein n denotes a natural number; extracting n face regions from the n images, respectively; and reconstructing a viewpoint-based face image based on the n face regions.

Claims

1. A method of reconstructing a three-dimensional (3D) face from images captured by a plurality of stereo cameras, the method comprising:

simultaneously acquiring n images of a target by controlling the plurality of stereo cameras in response to an image acquirement request, wherein n denotes a natural number equal to or greater than 4;

extracting n face regions from the n images, respectively;

reconstructing a viewpoint-based face image based on the n face regions;

determining an estimated error point in the viewpoint-based face image based on a 3D coordinate value or a disparity value of a point in the viewpoint-based face image that exceed respective predetermined error ranges;

removing the estimated error point from the viewpoint-based face image; and

performing matching processing on the viewpoint-based face image from which the estimated error point is removed,

wherein each stereo camera comprises a plurality of digital single-lens reflex (DSLR) cameras.

2. The method of claim 1, wherein the reconstructing comprises:

generating a disparity map about a stereo camera used to acquire an image; and

generating a 3D coordinate value at which a face region is positioned on the viewpoint-based face image, using the disparity map.

3. The method of claim 1, wherein the removing of the estimated error point comprises:

calculating a correspondence point of a predetermined point within a first face image and the disparity value of the correspondence point, from a second face image having a viewpoint different from a viewpoint of the first face image;

comparing the disparity value of the correspondence point with a disparity value of a neighboring correspondence point identified from the second face image; and

removing the correspondence point from the second face image when a difference between the disparity value of the correspondence point and the disparity value of the neighboring correspondence point exceeds the predetermined error range.

4. The method of claim 1, wherein the removing of the estimated error point comprises:

identifying a point having the disparity value or the 3D coordinate value exceeding the respective predetermined error ranges from the viewpoint-based face image, using face shape information of the target; and

removing the identified point from the viewpoint-based face image.

5. The method of claim 1, wherein the performing of the matching processing comprises:

merging the viewpoint-based face images into a single face image;

generating a texture map about the merged face image using visual field portions and a non-visual field portions of the plurality of stereo cameras; and

processing the entire tone of the generated texture map to be matched.

6. The method of claim 5, wherein the merging comprises merging an overlapping region between the viewpoint-abased face images to be merged, using an average value.

7. The method of claim 1, wherein the acquiring comprises:

controlling the plurality of stereo cameras to simultaneously capture an image of the target by generating a synchronization signal for each of the plurality of stereo cameras; and

storing and maintaining, in a memory, each captured image in association with identification information.

8. The method of claim 1, wherein the image acquirement request is generated through a hardware method using wired/wireless equipment or a software method using a button click on a personal computer (PC) screen.

9. The method of claim 1, wherein the extracting comprises:

modeling a background region included in an image, based on a guideline for each stereo camera; and

extracting a face region by removing a first background portion from a region acquired through modeling.

10. The method of claim 9, wherein the extracting of the face region further comprises:

further removing a second background portion included in a remaining region from which the first background portion is removed, using face shape information of the target;

or

extracting the face region by adding, to the remaining region, a face portion included in the removed first background portion.

11. The method of claim 1, further comprising:

generating a mask image of the extracted face region.

12. An apparatus for reconstructing a three-dimensional (3D) face from images captured by a plurality of stereo cameras, the apparatus comprising a non-transitory computer-readable medium which, when executed by the computer, performs the following method:

simultaneously acquiring n images of a target by controlling the plurality of stereo cameras in response to an image acquirement request, wherein n denotes a natural number equal to or greater than 4;

extracting n face regions from the n images, respectively;

reconstructing a viewpoint-based face image based on the n face regions;

determining an estimated error point in the viewpoint-based face image based on a 3D coordinate value or a disparity value of a point in the viewpoint-based face image that exceed respective predetermined error ranges;

removing the estimated error point from the viewpoint-based face image; and

performing matching processing on the viewpoint-based face image from which the estimated error point is removed,

wherein each stereo camera comprises a plurality of digital single-lens reflex (DSLR) cameras.