Apparatus and method for reconstructing three dimensional faces based on multiple cameras

• US 9,189,857 B2
• Filed: 05/06/2013
• Issued: 11/17/2015
• Est. Priority Date: 05/11/2012
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Yoon, Seung-Uk, Kim, Kap-Kee, Hwang, Bon-Woo, Lee, Ji-Hyung, Koo, Bon-Ki
• Primary Examiner(s): Heidemann, Jason
• Application Number:
  US13/888,333
• Publication Number:
  US 20130301906A1
• Time in Patent Office: 925 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/529 from texture
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

Disclosed herein are an apparatus and method for reconstructing a three-dimensional (3D) face based on multiple cameras. The apparatus includes a multi-image analysis unit, a texture image separation unit, a reconstruction image automatic synchronization unit, a 3D appearance reconstruction unit, and a texture processing unit. The multi-image analysis unit determines the resolution information of images received from a plurality of cameras, and determines whether the images have been synchronized with each other. The texture image separation unit separates a texture processing image by comparing the resolutions of the received images. The reconstruction image automatic synchronization unit synchronizes images that are determined to be asynchronous images by the multi-image analysis unit. The 3D appearance reconstruction unit computes the 3D coordinate values of the synchronized images, and reconstructs a 3D appearance image. The texture processing unit reconstructs a 3D image by mapping the texture processing image to the 3D appearance image.

Claims

1. An apparatus for reconstructing three-dimensional (3D) faces based on multiple cameras, comprising:

a multi-image analysis unit configured to determine resolution information of images received from a plurality of cameras, and to determine whether the images have been synchronized with each other;

a texture image separation unit configured to separate a texture processing image by comparing resolutions of the received images;

a reconstruction image automatic synchronization unit configured to synchronize images that are determined to be asynchronous images by the multi-image analysis unit;

a 3D appearance reconstruction unit configured to compute 3D coordinate values of the synchronized images, and to reconstruct a 3D appearance image; and

a texture processing unit configured to reconstruct a 3D image by mapping the texture processing image to the 3D appearance image,

wherein the reconstruction image automatic synchronization unit comprises;

a plurality of image buffers configured to divide frames of the images determined to be asynchronous images by a predetermined time interval and then store the divided asynchronous images; and

a synchronization checking unit configured to detect a time interval of a frame whose synchronization is the highest of the images determined to be asynchronous images and then takes an image in the time interval as a synchronous image.

2. The apparatus of claim 1, wherein the reconstruction image automatic synchronization unit operates in different modes depending on whether a depth image is included in the received images.

3. The apparatus of claim 2, wherein the 3D appearance reconstruction unit computes 3D coordinates of an object using depth information of a depth image received from a depth image camera, and then reconstructs the 3D appearance image.

4. The apparatus of claim 2, wherein the 3D appearance reconstruction unit triangulates entire image regions based on boundary surfaces of an object in color images received from a plurality of color image cameras, and then reconstructs the 3D appearance image.

5. The apparatus of claim 1, wherein the texture image separation unit separates an image whose resolution is highest of those of the received images as the texture processing image.

6. The apparatus of claim 1, wherein the multi-image analysis unit comprises:

an image resolution comparison unit configured to analyze resolutions of the received images, and to compare the resolutions of the images;

an image information classification unit configured to classify the received images into color images or depth images; and

a synchronization checking unit configured to convert an image classified as a color image by the image information classification unit into a grayscale image, and to determine whether the grayscale image and the remaining images have been synchronized with each other.

7. The apparatus of claim 1, wherein the image buffers are classified into one or more color image buffers for processing color images and one or more image buffers for processing depth images, and a number of image buffers is equal to a number of cameras that received images.

8. A method of reconstructing a 3D face using an apparatus for reconstructing 3D faces based on multiple cameras, comprising:

receiving images from a plurality of cameras;

analyzing types and resolutions of the received images;

separating a texture processing image by comparing the resolutions of the analyzed images;

checking whether the received images except the separated image have been synchronized with each other;

synchronizing images determined to be asynchronous images with each other;

computing 3D coordinate values for the synchronized images and reconstructing a 3D appearance image; and

mapping the texture processing image to the 3D appearance image,

wherein checking comprises;

converting a color image into a grayscale image when the color image is included in the received images except the separated image;

computing an insufficiency level of synchronization between the grayscale image and the remaining images; and

determining whether the insufficiency level of synchronization is equal to or higher than a predetermined allowable reference value, and determining the images to be asynchronous images when the insufficiency level of synchronization is equal to or higher than the predetermined allowable reference value.

9. The method of claim 8, wherein separating comprises separating an image whose resolution is the highest of those of the analyzed images as the texture processing image.

10. The method of claim 8, wherein mapping comprises reconstructing a 3D face image by mapping the texture processing image to the 3D appearance image using step size information of images that are used as the texture processing image and the 3D appearance image.

11. The method of claim 8, wherein analyzing is performed in different 3D appearance image reconstruction modes depending whether a depth image is included in the received images.

12. The method of claim 11, wherein computing and reconstructing comprises:

computing 3D coordinates of an object using depth information of a depth image received from a depth image camera; and

projecting the 3D coordinates onto a location of a color image camera, determining a relation of correspondence between the depth image and a color image, and reconstructing the 3D appearance image.

13. The method of claim 11, wherein computing and reconstructing comprises:

triangulating entire image regions based on boundary surfaces of an object in color images received from a plurality of color image cameras;

computing differences by comparing color values of the images for triangles created by the triangulation; and

computing 3D coordinates for the differences using camera information and then reconstructing the 3D appearance image.

14. A method of reconstructing a 3D face using an apparatus for reconstructing 3D faces based on multiple cameras, comprising:

receiving images from a plurality of cameras;

analyzing types and resolutions of the received images;

separating a texture processing image by comparing the resolutions of the analyzed images;

checking whether the received images except the separated image have been synchronized with each other;

synchronizing images determined to be asynchronous images with each other;

computing 3D coordinate values for the synchronized images and reconstructing a 3D appearance image; and

mapping the texture processing image to the 3D appearance image,

wherein synchronizing comprises;

dividing frames of the images determined to be the asynchronous images by a predetermined time interval;

storing the images determined to be asynchronous images in a plurality of image buffers, respectively; and

detecting a time interval of a frame whose synchronization is the highest of the images determined to be asynchronous images and then taking an image in the time interval as a synchronous image.

15. The method of claim 14, wherein the image buffers are classified into one or more color image buffers for processing color images and one or more image buffers for processing depth images, and a number of image buffers is equal to a number of cameras that received images.