Apparatus and method for creating three-dimensional caricature

• US 7,535,469 B2
• Filed: 05/05/2003
• Issued: 05/19/2009
• Est. Priority Date: 05/03/2002
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Park, Jun Ho, Hwang, Eui Hyeon, Kim, Dong Kwan, Kim, Jay Woo, Park, Dong Ryeol
• Primary Examiner(s): Chauhan; Ulka
• Assistant Examiner(s): PRENDERGAST, ROBERTA D
• Application Number:
  US10/428,818
• Publication Number:
  US 20030206171A1
• Time in Patent Office: 2,206 Days
• Field of Search: 345/419, 345/441, 345/582, 345/646, 345/647, 345/473, 345/420, 345/423, 382/154, 382/103, 382/115, 382/117, 382/118, 382/190, 382/195
• US Class Current: 345/419
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching

Abstract

The present invention relates to an apparatus and method for creating a three-dimensional caricature, and more particularly, to an apparatus and method for creating a three-dimensional caricature having a high degree of similarity to user'"'"'s face by precisely extracting feature point information on facial features and modifying a three-dimensional polygon face basic model based on the feature point information using an ASM technique specialized for faces. The method comprises the steps of detecting the positions of eyes from an input facial image and normalizing the size of the facial image, detecting each initial position for facial features from the normalized facial image, loading ASMs in the normalized facial image so as to correspond to the detected initial position and then extracting feature points for the facial features, and creating a three-dimensional polygon face caricature by modifying the three-dimensional polygon face basic model according to coordinate values of the extracted feature points.

Claims

1. An apparatus for creating a three-dimensional caricature based on a user'"'"'s input facial image, comprising:

a memory unit for storing ASMs (Active Shape Models) required to create the three-dimensional caricature, and three-dimensional polygon face basic models;

a pre-processing unit for detecting positions of eyes from the input facial image and normalizing the size of the facial image, the pre-processing unit calculating deviation values for eye candidate regions, calculating an error rate for the eye candidate regions by assigning a weight to each of the deviation values and summing the deviation values together, and determining whether the error rate for the eye candidate regions is within an allowable range;

a facial feature initial position-detecting unit for detecting each initial position for facial features from the normalized facial image;

an ASM loading unit for loading the ASMs stored in the memory unit in the normalized facial image so as to correspond to the detected initial position;

an ASM adapting unit for adapting the loaded ASMs so as to extract feature points for the facial features; and

a caricature-creating unit for creating a three-dimensional polygon face caricature by loading the three-dimensional polygon face basic model from the memory unit and then modifying the loaded three-dimensional polygon face basic model according to coordinate values of the feature points extracted through the ASM adapting unit.

2. The apparatus as claimed in claim 1, wherein the pre-processing unit includes:

a desaturing unit for, if the input facial image is a color image, converting the color image into a black and white image;

a facial region-detecting unit for detecting a facial region from the input facial image by using a Gabor filter response;

an eye-detecting unit for detecting positions of eyes from the detected facial region through the facial region-detecting unit; and

a face-normalizing unit for normalizing the size of face region such that the face region can be mapped in a one to one ratio with the three-dimensional polygon face basic model stored in the memory unit based on the positions of eyes detected through the eye-detecting unit.

3. The apparatus as claimed in claim 2, wherein the eye-detecting unit includes:

a binarization unit for performing binarization for the input facial image on the basis of a threshold;

a grouping unit for grouping black pixels in a region corresponding to eyebrows and eyes when the binary coded facial image is input from the binarization unit;

a labeling unit for sequencing a plurality of groups by labeling the plurality of groups detected through the grouping unit and then extracting regions having the same label for isolated regions; and

a decision unit for setting eyebrow candidate regions and eye candidate regions among the isolated regions extracted by the labeling unit and then determining whether the set eye candidate regions correspond to eyes.

4. The apparatus as claimed in claim 3, wherein the decision unit performs the binarization while varying the threshold until all of the eyebrow candidate regions and the eye candidate are detected.

5. The apparatus as claimed in claim 1, wherein the facial feature initial position-detecting units detects each initial position for the facial features by applying a Canny filter to the normalized facial image.

6. The apparatus as claimed in claim 1, wherein the memory unit further stores data on shape variation characteristics of the ASMs, and the ASM adapting unit extracts feature points for eyebrow, eye, nose, lip or face contours from the normalized facial image by using the data on shape variation characteristics of the ASMs stored in the memory unit.

7. The apparatus as claimed in claim 1, wherein the ASM adapting unit limits in advance search regions for the extraction of the feature points of the facial features when the feature points are extracted.

8. The apparatus as claimed in claim 1, wherein the memory unit further stores a two-dimensional face texture for texturing the three-dimensional polygon face basic model, and the caricature-creating unit maps the two-dimensional face texture stored in the memory unit into the three-dimensional polygon face basic model.

9. The apparatus as claimed in claim 1, wherein the memory unit further stores a three-dimensional body model required to create a three-dimensional body caricature, and the caricature-creating unit creates the three-dimensional body caricature based on the three-dimensional body model stored in the memory unit.

10. The apparatus as claimed in claim 1, wherein, when a side facial image is input,

the pre-processing unit detects an end point of the nose and eyebrows from the input side facial image and then normalizes the side facial image based on the detected end point of the nose and the detected eyebrows;

the facial feature initial position-detecting unit detects each side initial position for facial features from the normalized side facial image;

the ASM loading unit loads side ASMs stored in the memory unit in the normalized side facial image so as to correspond to the detected side initial position;

the ASM adapting unit adapts the loaded side ASMs so as to extract coordinate values of feature points for the facial features from the normalized side facial image; and

the caricature-creating unit creates a three-dimensional polygon face caricature by modifying the three-dimensional polygon face basic model according to the coordinate values of the feature points extracted through the ASM adapting unit.

11. The apparatus as claimed in claim 1, further comprising a projecting unit for obtaining a two-dimensional face caricature by projecting the three-dimensional polygon face caricature created by the caricature-creating unit onto a two-dimensional plane.

12. The apparatus as claimed in claim 1, further comprising a caricature transmitting/receiving unit for transmitting/receiving coordinate values and memory reference values of the feature points extracted from the three-dimensional polygon face caricature created through the caricature-creating unit.

13. The apparatus as claimed in claim 12, wherein, when the coordinate values and memory reference values of the feature points extracted from the three-dimensional polygon face caricature are received from the caricature transmitting/receiving unit, the caricature-creating unit loads a three-dimensional polygon face basic model and a two-dimensional face texture corresponding to the received memory reference values from the memory unit, maps the two-dimensional face texture onto the three-dimensional polygon face basic model, and then reconstructs a three-dimensional polygon face caricature by modifying the three-dimensional polygon face basic model according to the coordinate values of the received feature points.

14. The apparatus of claim 1 wherein said ASM adapting unit performs the function of adapting said ASM by:

connecting feature points of the ASMs by a straight line;

defining a search region normal to said straight line at the midpoint thereof, said search region being limited in size;

searching within said search region for a pixel having a maximal correlation with a basic learning value of the ASM and using said pixel as a new midpoint; and

changing the positions of said feature points according to said new midpoint.

15. The apparatus of claim 14 wherein said limit of the size of said search region is determined by the distances between the facial features on said facial image.

16. A method for creating a three-dimensional caricature based on a user'"'"'s input facial image, comprising the steps of:

performing pre-processing by detecting positions of eyes from the input facial image and normalizing the size of the facial image, the pre-processing unit calculating deviation values for eye candidate regions, calculating an error rate for the eye candidate regions by assigning a weight to each of the deviation values and summing the deviation values together, and determining whether the error rate for the eye candidate regions is within an allowable range;

detecting each initial position for facial features from the normalized facial image;

loading ASMs in the normalized facial image so as to correspond to the detected initial position and then adapting the loaded ASMs so as to extract feature points for the facial features; and

creating a three-dimensional polygon face caricature by loading a three-dimensional polygon face basic model and then modifying the loaded three-dimensional polygon face basic model according to coordinate values of the feature points extracted in the ASM adapting step.

17. The method as claimed in claim 16, wherein the step of performing the pre-processing comprises:

if the input facial image is a color image, converting the color image into a black and white image;

detecting a facial region from the input facial image by using a Gabor filter response;

detecting the positions of eyes from the detected facial region; and

normalizing the size of the facial region such that the facial region can be mapped in a one to one ratio with the three-dimensional polygon face basic model based on the detected positions of the eyes.

18. The method as claimed in claim 17, wherein the step of detecting the positions of the eyes comprises:

performing binarization for the input facial image on the basis of a threshold;

grouping black pixels in the binary coded facial image;

sequencing a plurality of groups by labeling the plurality of groups detected in the step of grouping and then extracting regions having the same label for isolated regions; and

setting eyebrow candidate regions and eye candidate regions among the extracted isolated regions and then detecting midpoints of both eyes from the set eye candidate regions.

19. The method as claimed in claim 18, wherein the step of performing the binarization comprises the steps of setting a range of threshold variation according to brightness of the input facial image, setting an initial value for threshold within the set range of threshold variation, and then performing the binarization for the input facial image on the basis of the set initial value of the threshold.

20. The method as claimed in claim 18, wherein the grouping step comprises the step of connecting the black pixels together when a gap between black pixels spaced in a horizontal direction is calculated and the calculated gap is less than a predetermined distance.

21. The method as claimed in claim 18, wherein the labeling step comprises the step of removing groups having sizes less than a predetermined size among the plurality of groups detected in the grouping step, which are regarded as noise.

22. The method as claimed in claim 18, further comprising the step of, if there are no pair of eyebrow candidate regions and pair of eye candidate regions among the extracted isolated regions, performing the binarization while increasing the threshold until both a pair of eyebrow candidate regions and a pair of eye candidate regions are detected.

23. The method as claimed in claim 16, wherein the step of detecting each initial position comprises the step of determining an initial position of each facial feature by applying a Canny filter to the normalized facial image.

24. The method as claimed in claim 16, wherein the step of extracting the feature points comprises the step of extracting the feature points for eyebrow, eye, nose, lip or face contours from the normalized facial image by using data on shape variation characteristics of ASMs.

25. The method as claimed in claim 16, wherein search regions of extraction of the feature points for facial features are limited in advance when the feature points are extracted.

26. The method as claimed in claim 16, further comprising the step of mapping a two-dimensional face texture onto the three-dimensional polygon face basic model.

27. The method as claimed in claim 16, further comprising the step of creating the three-dimensional body caricature based on a three-dimensional body model.

28. The method as claimed in claim 16, further comprising the steps of, when a side facial image is input,

detecting an end point of nose and eyebrows from the input side facial image and then normalizing the side facial image based on the detected end point of nose and the detected eyebrows;

detecting each side initial position for facial features from the normalized side facial image;

loading side ASMs in the normalized side facial image so as to correspond to the detected side initial position;

adapting the loaded side ASMs so as to extract coordinate values of feature points for the facial features from the normalized side facial image; and

creating a three-dimensional polygon face caricature by modifying the loaded three-dimensional polygon face basic model according to the coordinate values of the feature points extracted in the adapting step.

29. The method as claimed in claim 16, further comprising the step of obtaining a two-dimensional face caricature by projecting the three-dimensional polygon face caricature onto a two-dimensional plane.

30. The method as claimed in claim 16, further comprising the step of transmitting coordinate values and memory reference values of the feature points extracted from the three-dimensional polygon face caricature.

31. The method as claimed in claim 30, further comprising the steps of, when the coordinate values and memory reference values of the feature points extracted from the three-dimensional polygon face caricature are received,

loading a three-dimensional polygon face basic model and a two-dimensional face texture corresponding to the received memory reference values;

mapping the two-dimensional face texture onto the three-dimensional polygon face basic model; and

reconstructing a three-dimensional polygon face caricature by modifying the three-dimensional polygon face basic model according to the coordinate values of the received feature points.

32. The method as claimed in claim 16 wherein said step of adapting the loaded ASMs comprises the steps of:

connecting feature points of the ASMs by a straight line;

defining a search region normal to said straight line at the midpoint thereof, said search region being limited in size;

searching within said search region for a pixel having a maximal correlation with a basic learning value of the ASM and using said pixel as a new midpoint; and

changing the positions of said feature points according to said new midpoint.

33. The method as claimed in claim 32 wherein said limit of the size of said search region is determined by the distances between the facial features on said facial image.

34. A method for creating a three-dimensional caricature based on a user'"'"'s input facial image, comprising the steps of:

if the input facial image is a color image, converting the color image into a black and white image;

detecting a facial region from the input facial image by using a Gabor filter response;

performing binarization for the input facial image on the basis of a threshold;

grouping black pixels in the binary coded facial image;

sequencing a plurality of groups by labeling the plurality of groups detected in the step of grouping and then extracting regions having the same label for isolated regions;

setting eyebrow candidate regions and eye candidate regions among the extracted isolated regions;

comparing information on the eye candidate regions with reference values stored in advance and then calculating deviation values for the eye candidate regions;

calculating an error rate for the eye candidate regions by assigning a weight to each of the deviation values and summing the deviation values together;

determining whether the error rate for the eye candidate regions is within an allowable range;

normalizing the size of the facial region such that the facial region can be mapped in a one to one ratio with the three-dimensional polygon face basic model based on the detected positions of the eyes;

detecting each initial position for facial features from the normalized facial image;

loading ASMs in the normalized facial image so as to correspond to the detected initial position and then adapting the loaded ASMs so as to extract feature points for the facial features; and

creating a three-dimensional polygon face caricature by loading a three-dimensional polygon face basic model and then modifying the loaded three-dimensional polygon face basic model according to coordinate values of the feature points extracted in the ASM adapting step.

35. The method as claimed in claim 34, wherein the reference values stored in advance include values for the size, ratio of width to length, vertical position and horizontal position of the eyes.

36. The method as claimed in claim 34, wherein the information on eye candidate regions is compared with the reference values for a corresponding threshold when deviation values for the eye candidate regions are calculated.

37. The method as claimed in claim 34, wherein, if only a pair of eyebrow candidate regions among the extracted isolated regions are detected, the detected pair of eyebrow candidate regions are set to eye candidate regions, and then an error rate for the eye candidate regions is calculated.

38. The method as claimed in claim 34, wherein, if only one eyebrow candidate region among the extracted isolated regions is detected, the detected eyebrow candidate region is set to an eye candidate region, a pair of eyes are created based on the eye candidate region, and then an error rate for the created pair of eyes is calculated.

39. An apparatus for creating a three-dimensional caricature based on a user'"'"'s input facial image, comprising:

a memory unit for storing ASMs (Active Shape Models) required to create the three-dimensional caricature, and three-dimensional polygon face basic models that are based on anthropological standard data reflecting racial traits;

a pre-processing unit for detecting positions of eyes from the input facial image and normalizing the size of the facial image, the pre-processing unit calculating deviation values for eye candidate regions, calculating an error rate for the eye candidate regions by assigning a weight to each of the deviation values and summing the deviation values together, and determining whether the error rate for the eye candidate regions is within an allowable range;

a facial feature initial position-detecting unit for detecting each initial position for facial features from the normalized facial image;

an ASM loading unit for loading the ASMs stored in the memory unit in the normalized facial image so as to correspond to the detected initial position;

an ASM adapting unit for adapting the loaded ASMs so as to extract feature points for the facial features; and

a caricature-creating unit for creating a three-dimensional polygon face caricature by loading the three-dimensional polygon face basic model from the memory unit and then modifying the loaded three-dimensional polygon face basic model according to coordinate values of the feature points extracted through the ASM adapting unit.