METHOD AND APPARATUS FOR RECONSTRUCTING 3D FACE WITH STEREO CAMERA

• US 20150228081A1
• Filed: 01/30/2015
• Published: 08/13/2015
• Est. Priority Date: 02/10/2014
• Status: Active Grant

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: KIM, Kap Kee, HWANG, Bon Woo, LIM, Seong Jae, JUN, Hye Ryeong, CHOI, Jin Sung, KOO, Bon Ki, YOON, Seung Uk
• Granted Patent: US 10,043,278 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Provided is a method and apparatus for reconstructing a three-dimensional (3D) face based on a stereo camera, the method including: acquiring n images of a target by controlling a plurality of stereo cameras in response to an image acquirement request, wherein n denotes a natural number; extracting n face regions from the n images, respectively; and reconstructing a viewpoint-based face image based on the n face regions.

Claims

1. A method of reconstructing a three-dimensional (3D) face based on a stereo camera, the method comprising:

acquiring n images of a target by controlling a plurality of stereo cameras in response to an image acquirement request, wherein n denotes a natural number;

extracting n face regions from the n images, respectively; and

reconstructing a viewpoint-based face image based on the n face regions.

2. The method of claim 1, wherein the reconstructing comprises:

generating a disparity map about a stereo camera used to acquire an image; and

generating a 3D coordinate value at which a face region is positioned on the viewpoint-based face image, using the disparity map.

3. The method of claim 1, further comprising:

removing an estimated error point from the viewpoint-based face image; and

performing matching processing on the viewpoint-based face image from which the estimated error point is removed.

4. The method of claim 3, wherein the removing of the estimated error point comprises:

calculating a correspondence point of a predetermined point within a first face image and a disparity value of the correspondence point, from a second face image having a viewpoint different from a viewpoint of the first face image;

comparing the disparity value of the correspondence point with a disparity value of a neighboring correspondence point identified from the second face image; and

removing the correspondence point from the second face image when a difference between the disparity value of the correspondence point and the disparity value of the neighboring correspondence point exceeds a predetermined error range.

5. The method of claim 3, wherein the removing of the estimated error point comprises:

identifying a point having a disparity value or a 3D coordinate value exceeding a predetermined error range from the viewpoint-based face image, using face shape information of the target; and

removing the identified point from the viewpoint-based face image.

6. The method of claim 3, wherein the performing of the matching processing comprises:

merging the viewpoint-based face images into a single face image;

generating a texture map about the merged face image using visual field portions and a non-visual field portions of the plurality of stereo cameras; and

processing the entire tone of the generated texture map to be matched.

7. The method of claim 6, wherein the merging comprises merging an overlapping region between the viewpoint-abased face images to be merged, using an average value.

8. The method of claim 1, wherein the acquiring comprises:

controlling the plurality of stereo cameras to simultaneously capture an image of the target by generating a synchronization signal for each of the plurality of stereo cameras; and

storing and maintaining, in a memory, each captured image in association with identification information.

9. The method of claim 1, wherein the image acquirement request is generated through a hardware method using wired/wireless equipment or a software method using a button click on a personal computer (PC) screen.

10. The method of claim 1, wherein the extracting comprises:

modeling a background region included in an image, based on a guideline for each stereo camera; and

extracting a face region by removing a first background portion from a region acquired through modeling.

11. The method of claim 10, wherein the extracting of the face region further comprises:

further removing a second background portion included in a remaining region from which the first background portion is removed, using face shape information of the target;

or

extracting the face region by adding, to the remaining region, a face portion included in the removed first background portion.

12. The method of claim 1, further comprising:

generating a mask image of the extracted face region.

13. The method of claim 1, wherein each stereo camera comprises a plurality of digital single-lens reflex (DSLR) cameras.

14. An apparatus for reconstructing a three-dimensional (3D) face based on a stereo camera, the apparatus comprising:

an acquirer configured to acquire n images of a target by controlling a plurality of stereo cameras in response to an image acquirement request, wherein n denotes a natural number;

an extractor configured to extract n face region from the n images, respectively; and

a re-constructer configured to reconstruct a viewpoint-based face image based on the n face regions.

15. The apparatus of claim 14, wherein the re-constructer is configured to generate a disparity map about a stereo camera used to acquire an image, and to generate a 3D coordinate value at which a face region is positioned on the viewpoint-based face image, using the disparity map.

16. The apparatus of claim 14, further comprising:

a matcher configured to remove an estimated error point from the viewpoint-based face image, and to perform matching processing on the viewpoint-based face image from which the estimated error point is removed.

17. The apparatus of claim 16, wherein the matcher is configured to calculate a correspondence point of a predetermined point within a first face image and a disparity value of the correspondence point, from a second face image having a viewpoint different from a viewpoint of the first face image, to compare the disparity value of the correspondence point with a disparity value of a neighboring correspondence point identified from the second face image, and to remove the correspondence point from the second face image when a difference between the disparity value of the correspondence point and the disparity value of the neighboring correspondence point exceeds a predetermined error range.

18. The apparatus of claim 16, wherein the matcher is configured to identify a point having a disparity value or a 3D coordinate value exceeding a predetermined error range from the viewpoint-based face image, using face shape information of the target, and to remove the identified point from the viewpoint-based face image.

19. The apparatus of claim 16, wherein the matcher is configured to merge the viewpoint-based face images into a single face image, to generate a texture map about the merged face image using visual field portions and non-visual field portions of the plurality of stereo cameras, and to process the entire tone of the generated texture map to be matched.

20. The apparatus of claim 14, wherein the acquirer is configured to control the plurality of stereo cameras to simultaneously capture an image of the target by generating a synchronization signal for each of the plurality of stereo cameras;

and to store and maintain, in a memory, each captured image in association with identification information.