Method and apparatus for image-based photorealistic 3D face modeling

• US 7,835,568 B2
• Filed: 08/27/2004
• Issued: 11/16/2010
• Est. Priority Date: 08/29/2003
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Zhang, Hui, Vezhnevets, Vladimir, Choh, Heui-Keun, Park, In-Kyu
• Primary Examiner(s): Le; Vu
• Assistant Examiner(s): VANCHY JR, MICHAEL J
• Application Number:
  US10/927,241
• Publication Number:
  US 20050063582A1
• Time in Patent Office: 2,272 Days
• Field of Search: 382/154
• US Class Current: 382/154
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes

Abstract

An apparatus and method for image-based 3D photorealistic head modeling are provided. The method for creating a 3D photorealistic head model includes: detecting frontal and profile features in input frontal and profile images; generating a 3D head model by fitting a 3D genetic model using the detected facial features; generating a realistic texture from the input frontal and profile images; and mapping the texture onto the 3D head model. In the apparatus and method, data obtained using a relatively cheap device, such as a digital camera, can be processed in an automated manner, and satisfactory results can be obtained even from imperfect input data. In other words, facial features can be extracted in an automated manner, and a robust “human-quality” face analysis algorithm is used.

Claims

1. A method for creating a 3D photorealistic head model, the method being executed by a processor and comprising:

(a) detecting frontal and profile features in input frontal and profile images, the detecting frontal and profile features comprising;

detecting a face region, the detecting a face region comprising initializing the largest skin-colored connected component as the face region, and when there are two largest connected components having similar sizes, the initializing includes selecting one of the two largest connected components that is closer to the center of the image as the face region, and

detecting a profile region by classifying pixels into skin color pixels and non-skin color pixels and selecting the largest skin-colored connected component as the profile region;

(b) generating a 3D head model by fitting a 3D generic model using the detected face region and the detected profile region;

(c) generating a realistic texture from the input frontal and profile images; and

(d) mapping the texture onto the 3D head model.

2. The method of claim 1, wherein (b) generating the 3D head model further comprises generating depth information from the frontal image and a near-frontal image, the generic model is fitted using the facial features to determine the 3D shape of the particular face, and the generated 3D head model is fitted using the depth information.

3. The method of claim 2, wherein the generating the depth information from the frontal and near-frontal images comprises:

determining features points in the frontal image;

performing model-based Lucas-Kanade tracking on the feature points to find corresponding points in the near-frontal image;

obtaining a dense disparity map by data interpolation using matching correspondence; and

converting the disparity map into the depth information.

4. The method of claim 3, wherein the converting the disparity map into the depth information comprises:

obtaining a difference value between maximum and minimum disparity values;

determining a coefficient by dividing the difference value by a head width and multiplying the divided result by a predetermined constant; and

obtaining the depth value of a pixel by multiplying the coefficient by a value obtained by subtracting the minimum disparity value from a measured disparity value.

5. The method of claim 1, wherein (c) further comprises blending an artificial texture.

6. The method of claim 1, wherein (a) detecting the frontal features further comprises:

detecting a skin region;

detecting eye bounding boxes;

detecting irises; and

normalizing the frontal image.

7. The method of claim 6, wherein the detecting the skin region is performed using a pre-trained maximum likelihood Bayesian classifier, which classifies image pixels into skin pixels and non-skin pixels in a normalized r-g chrominance color space.

8. The method of claim 7, wherein the detecting the face region further comprises:

initializing and evolving an ellipse by displacing boundary points of the ellipse based on the initialized face region until the ellipse is stabilized; and

determining a bounding box of the stabilized ellipse as the face region.

9. The method of claim 8, wherein (ii) initializing and evolving the ellipse comprises:

defining a rectangular probe centered at a boundary point of the ellipse based on the initialized face region;

calculating the number of skin pixels that lie inside both the probe and the ellipse;

moving the boundary point of the ellipse a predetermined distance inwards if the calculated number of skin pixels is less than a predetermined threshold;

calculating the number of skin pixels that lie inside the probe and outside ellipse if the calculated number of skin pixels is not less than the predetermined threshold value; and

moving the boundary point of the ellipse a predetermined distance outwards if the calculated number of skin pixels that lie inside the prove and outside ellipse is greater than a predetermined threshold.

10. The method of claim 7, wherein the detecting the eye bounding boxes comprises:

generating a blurred image by low-pass filtering;

obtaining an image by subtracting the blurred image from the original image;

generating a new image by calculating the mean value of each pixel and its neighboring pixels in the obtained image and binarizing the new image using a predetermined threshold value;

finding connected segments as eye candidates; and

determining the eye bounding boxes by evaluating the eye candidates in terms of shape, size, and relative position.

11. The method of claim 6, wherein the detecting the irises comprises detecting the irises within the eye bounding boxes in a red channel image.

12. The method of claim 11 further comprising detecting the centers and radii of the irises, the detecting the centers and radii of the irises comprising:

comparing the number of pixels having maximum brightness values in each of the eye bounding boxes in the red channel image with a threshold value to determine whether a highlight exists;

initializing the center of mass of the darkest pixels as the center of the iris if no highlight is found;

initializing the center of a high contrast region near a highlight as the center of the iris if the highlight exists;

enlarging an initial circle at the center of the iris and measuring gradients of the pixels along the circle; and

enlarging the initial circle at the center of the iris and repeatedly measuring the gradients of the pixels along the circle; and

selecting a region having the largest gradient as the center of the iris and determining the radius of the iris.

13. The method of claim 6, wherein the normalizing the frontal image comprises rotating the frontal image such that the line between the centers of the irises becomes horizontal and scaling the frontal image to a predetermined resolution.

14. The method of claim 1, wherein (a) detecting the frontal features comprises at least one of:

detecting eye contours;

detecting eyebrows;

detecting a lips contour;

detecting a nose contour; and

detecting a chin and cheeks contour.

15. The method of claim 14, wherein the detecting the eye contours comprises approximating the eye contours using two polynomials for upper and lower eye contours, the approximating the eye contours comprising:

predetermining a search area according to the center and radius of the iris;

selecting pixels on each horizontal scan line in a red channel image in the search area, in which local minimums or brightness abruptly change, as left and right eyelid points;

selecting the leftmost and rightmost points among the remaining eyelid points as eye corners;

estimating an upper eyelid curve using points on a line connecting the eye corners by a minimum square method; and

estimating a lower eyelid curve using the selected eye corners and a lower point of the iris.

16. The method of claim 14, wherein the detecting the eyebrows comprises approximating the eyebrows using a polynomials that fits dark pixels in a predetermined area above the eye bounding box after adaptive binarization.

17. The method of claim 14, wherein the detecting the lips contour comprises detecting upper and lower lips, the detecting the upper and lower lips comprising:

initializing a mouth rectangle;

refining the mouth rectangle to be smaller by analyzing the first order and second order moments of non-skin color pixels;

generating a lip function image and calculating the probability that each pixel in the refined mouth rectangle belongs to a lip or skin;

initializing a lips contour by an ellipse by measuring the second-order central moments of pixels having large lip function values;

dynamically moving the lips contour by inward and outward forces; and

generating a polynomials expressing the lips contour by approximation using moved contour points.

18. The method of claim 17, wherein the outward force is a force moving contour points to the boundary between the lips and skin, and the inward force consists of an internal shaping force, which shapes the contour to be elliptical, and an internal smoothening force, which is determined by predetermined factors including a force suppressing contour points from being moved beyond the boundary.

19. The method of claim 14, wherein the detecting the nose contour comprises detecting nose sides, the detecting the nose sides comprising:

performing matching with a temperate having a predetermined shape on an edge image in a predetermined search area;

selecting among detected candidates having large matching values a pair of candidates that satisfy the requirement for symmetry and determining the selected pair of candidates as wing positions; and

expressing each of the nose sides using a polynominal that is fitted to detected nose wing templates and three points interpolated between the nose wing and eye corners with predetermined coefficients.

20. The method of claim 14, wherein the detecting the chin and cheeks contour comprises:

calculating a gradient vector field on a red channel image;

positioning an initial contour template in a face region defined by two polynominal curves crossing at a chin point;

expanding the chin and cheeks contour outward until the chin and cheek contour reaches strong edges along the tangents of the curves; and

stopping expanding the chin and cheeks contour when the chin and cheek contour has reached the strong edges along the tangents of the curves.

21. The method of claim 1, wherein (a) detecting the profile features further comprises:

detecting a profile curve;

detecting fiducial points; and

detecting an ear contour.

22. The method of claim 21, wherein, when there are two largest connected components having similar sizes, one of the two largest connected components that is closer to the center of the image is selected as the profile region.

23. The method of claim 21, wherein the detecting the profile curve comprises detecting the rightmost boundary of a face region in a user-specified bounding box defined from the frontal image as a profile curve.

24. The method of claim 21, wherein the detecting the ear contour comprises:

initializing the ear position by matching a curve template with the profile image; and

finding a more accurate ear boundary by deforming an initialized ear template using a contour following algorithm.

25. The method of claim 24, wherein the initializing the ear position comprises:

rotating the original image such that a segment between a nose bridge top and a chin point becomes vertical and scaling the length of the segment to a predetermined value;

defining an ear search area;

matching the border between the profile region and a hair region with the curve template to find corresponding sub-segments having similar shape from the two curves;

detecting segments having similar shapes by matching an edge detected using an edge detector and the ear template if the length of the matching segments is smaller than a predetermined threshold value; and

calculating a translation vector from the template sub-segment to the matching image points with a least square algorithm.

26. The method of claim 1, wherein the fitting the generic model in (b) comprises:

converting feature curves detected in the images into 2D points corresponding to features defined in the 3D generic model using the length ratios of feature points on a facial feature curve of the 3D generic model; and

fitting the 3D generic model by 3-step radial based function (RBF)-based morphing based on the correspondences between the images features and the feature points of the 3D generic model.

27. The method of claim 26, wherein the 3-step RBF-based morphing comprises:

setting X and Y coordinates of displaced model key points to corresponding image feature points and remaining Z coordinate as in the 3D generic model to perform RBF interpolation on the frontal features;

adjusting the scale of the profile image in consideration of the profile features and using scaled profile features as input for RBF interpolation while constraining their X coordinates to remain at their positions in the generic model; and

performing RBF interpolation on both frontal and profile features using the Z coordinates obtained in previous step.

28. The method of claim 27, wherein the 3-step RBF-based morphing is performed such that vertices of the generic mesh are displaced to corresponding 3D points acquired in a depth information recovery process.

29. The method of claim 1, wherein (c) generating the realistic texture comprises:

performing color correction on the input images to compensate for different lightening conditions or color balance variations between cameras;

generating frontal and profile textures by mapping the frontal and profile images onto a public UV plane based on the correspondence between 3D model vertices and projected 2D positions; and

combining the frontal and profile textures and an artificial texture using a multi-resolution spline algorithm to complete texturing.

30. An apparatus for creating a 3D photorealistic head model, the apparatus comprising:

a face feature detecting unit receiving input frontal and profile images, and including a face region initializing portion that initializes the largest skin-colored connected components as a face region, wherein when there are two largest connected components having similar sizes, the initializing includes selecting one of the two largest connected components that is closer to the center of the image as the face region, and

a profile feature detecting portion that detects profile features, and includes a profile region detecting portion that detects a profile region by classifying pixels into skin color pixels and non-skin color pixels and selecting the largest skin-colored connected component as the profile region;

a face model fitting unit generating a 3D head model by fitting a 3D generic model using the detected facial features;

a texture generating unit generating a realistic texture from the input frontal and profile images; and

a mapping unit mapping the texture onto the 3D head model generated in the face model fitting unit.

31. The apparatus of claim 30, wherein the face model fitting unit comprises a depth information generating portion extracting depth information from the frontal image and a near-frontal image and generates the 3D head model by fitting the generic model using the facial features detected in the face feature detecting unit and the depth information generated in the depth information generating portion.

32. The apparatus of claim 31, wherein the depth information generating unit comprises;

a feature point detecting portion determining feature points in the frontal image and finding corresponding feature points in the near-frontal image;

a disparity obtaining portion obtaining a dense disparity map by data interpolation using matching correspondence; and

a disparity-to-depth information converting portion converting the disparity map into the depth information.

33. The apparatus of claim 30, wherein the texture generating unit blends the realistic texture and an artificial texture.

34. The apparatus of claim 30, wherein the face feature detecting unit comprises:

A frontal feature detecting portion detecting the frontal features; and

a profile feature detecting portion detecting the profile features.

35. The apparatus of claim 34, wherein the frontal feature detecting portion comprises:

a skin region detecting portion detecting a skin region;

a face region detecting portion detecting a face region;

an eye bounding boxes detecting portion detecting eye bounding boxes;

an iris detecting portion detecting irises; and

a frontal image normalizing potion normalizing the frontal image.

36. The apparatus of claim 35, wherein the skin region detecting portion detects the skin region using a pre-trained maximum likelihood Bayesian classifier, which classifies image pixels into skin pixels and non-skin pixels in a normalized r-g chrominance color space.

37. The apparatus of claim 35, wherein the face region detecting unit comprises:

a the face region initializing portion;

an ellipse stabilizing portion initializing and evolving an ellipse by displacing boundary points of the ellipse based on the initialized face region until the ellipse is stabilized; and

a face region determining portion determining a bounding box of the stabilized ellipse as the face region.

38. The apparatus of claim 35, wherein the eye bounding boxes detecting portion comprises:

a blurred image generating portion generating a blurred image by low-pass filtering;

an image binarizing portion generating a new image by calculating the mean value of each pixel and neighboring pixels in an image obtained by subtracting the blurred image from the original image and binarizing the new image using a predetermined threshold value;

an eye candidate searching portion finding connected segments in the image binarized in the image binarizing portion as eye candidates; and

an eye bounding box determining portion determining the eye bounding boxes by evaluating the eye candidates in terms of shape, size, and relative position.

39. The apparatus of claim 35, wherein the iris detecting portion detects the irises within the eye bounding boxes in a red channel image.

40. The apparatus of claim 39, wherein the iris detecting portion comprises:

a highlight searching portion determining whether a highlight exists by comparing the number of pixels having maximum brightness values in each of the eye bounding boxes in the red channel image with a threshold value;

an iris center initializing portion initializing the center of mass of the darkest pixels as the center of the iris if no highlight is found and initializing the center of a high contrast region near a highlight as the center of the iris if the highlight exists;

an iris center/radius measuring portion repeatedly measuring gradients of the pixels along an initial circle at the center of the iris while enlarging the initial circle; and

an iris determining portion determining a region having the largest gradient as the center of the iris and the radius of the iris.

41. The apparatus of claim 35, wherein the frontal image normalizing portion normalizes the frontal image by rotating the frontal image such that the line between the centers of the irises becomes horizontal and scaling the frontal image to a predetermined resolution.

42. The apparatus of claim 34, wherein the frontal feature detecting portion further comprises at least one of an eye contours detecting portion, an eyebrows detecting portion, a lips contour detecting portion, a nose contour detecting portion, and a chin and cheeks contour detecting portion.

43. The apparatus of claim 42, wherein the eye contours detecting portion approximates the eye contours using two polynominals for upper and lower eye contours, the approximating the eye contours comprising:

predetermining a search area according to the center and radius of the iris;

selecting pixels on each horizontal scan line in a red channel image in the search area, in which local minimums or brightness abruptly change, as left and right eyelid points;

selecting the leftmost and rightmost points among the remaining eyelid points as eye corners;

estimating an upper eyelid curve using points on a line connecting the eye corners by a minimum square method; and

estimating a lower eyelid curve using the selected eye corners and a lower point of the iris.

44. The apparatus of claim 42, wherein the eyebrows detecting portion detects the eyebrows by approximation using a polynominal that fits dark pixels in a predetermined area above the eye bounding box after adaptive binarization.

45. The apparatus of claim 42, wherein the lips contour detecting portion detects upper and lower lips, the detecting the upper and lower lips comprising:

initializing a mouth rectangle;

refining the mouth rectangle to be smaller by analyzing the first order and second order moments of non-skin color pixels;

generating a lip function image and calculating the probability that each pixel in the refined mouth rectangle belongs to a lip or skin;

initializing a lips contour by an ellipse by measuring the second-order central moments of pixels having large lip function values;

dynamically moving the lips contour by inward and outward forces; and

generating a polynominal expressing the lips contour by approximation using moved contour points.

46. The apparatus of claim 42, wherein the nose contour detecting portion detects nose sides, the detecting the nose sides comprising:

performing matching with a temperate having a predetermined shape on an edge image in a predetermined search area;

selecting among detected candidates having large matching values a pair of candidates that satisfy the requirement for symmetry and determining the selected pair of candidates as wing positions; and

expressing each of the nose sides using a polynominal that is fitted to detected noise wing templates and three points interpolated between the nose wing and eye corners with predetermined coefficients.

47. The apparatus of claim 42, wherein the chin and cheeks contour detecting portion comprises:

a gradient calculating portion calculating a gradient vector field on a red channel image;

a chin and cheeks contour expanding portion positioning an initial contour template in a face region defined by two polynominal curves crossing at a chin point and expanding the chin and cheeks contour outward until the chin and cheek contour reaches strong edges along the tangents of the curves; and

a chin and cheeks contour determining portion stopping expanding the chin and cheeks contour when the chin and cheeks contour has reached the strong edges along the tangents of the curves.

48. The apparatus of claim 34, wherein the profile feature detecting portion comprises

a profile region detecting portion, a profile curve detecting portion,

a fiducial point detecting portion, and

an ear contour detecting portion.

49. The apparatus of claim 48, wherein the profile curve detecting portion detects the rightmost boundary of a face region in a user-specified bounding box defined from the frontal image as a profile curve.

50. The apparatus of claim 48, wherein the ear contour detecting portion comprises an ear contour by initializing the ear position by matching a curve template with the profile image and finding a more accurate ear boundary by deforming an initialized ear template using a contour following algorithm.

51. The apparatus of claim 30, wherein the generic model fitting unit comprises;

a 2D point converting portion converting feature curves detected in the images into 2D points corresponding to features defined in the 3D generic model using the length ratios of feature points on a facial feature curve of the 3D generic model; and

a mesh model fitting portion fitting the 3D generic model by 3-step radial based function (RBF)-based morphing based on the correspondences between the images features and the feature points of the 3D generic model.

52. The apparatus of claim 30, wherein the texture generating unit comprises:

an input image compensating portion compensating for different lightening conditions or color balance variations between cameras by performing color correction on the input images;

a frontal/profile texture generating portion generating frontal and profile textures by mapping the frontal and profile images onto a public UV plane based on the correspondence between 3D model vertices and projected 2D positions; and

a texturing completing portion combining the frontal and profile textures and an artificial texture using a multi-resolution spline algorithm to complete texturing.

53. A non-transitory computer readable storage medium having embodied thereon computer readable code to control at least one processing element in a computer to implement a method for creating a 3D photorealistic head model, the method comprising:

detecting frontal and profile features in input frontal and profile images, the detecting frontal and profile features comprising

detecting a face region, wherein the detecting a face region comprises initializing the largest skin-colored connected component as the face region, and wherein, when there are two largest connected components having similar sizes, the initializing includes selecting one of the two largest connected components that is closer to the center of the image as the face region;

detecting a profile region by classifying pixels into skin color pixels and non-skin color pixels and selecting the largest skin-colored connected component as the profile region;

generating a 3D head model by fitting a 3D genetic generic model using the detected face region and the detected profile region;

generating a realistic texture from the input frontal and profile images; and

mapping the texture onto the 3D head model.