Separating a Directional Lighting Variability In Statistical Face Modelling Based On Texture Space Decomposition

• US 20090003661A1
• Filed: 09/03/2008
• Published: 01/01/2009
• Est. Priority Date: 02/28/2007
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.), National University of Ireland Galway
• Original Assignee: FotoNation Vision Limited (Adeia Inc.), National University of Ireland Galway
• Inventors: Bacivarov, Ioana, Corcoran, Peter, Ionita, Mircea
• Granted Patent: US 8,224,039 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 10/7557 based on appearance, e.g. active appearance models [AAM]
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means

Abstract

A technique for determining a characteristic of a face or certain other object within a scene captured in a digital image including acquiring an image and applying a linear texture model that is constructed based on a training data set and that includes a class of objects including a first subset of model components that exhibit a dependency on directional lighting variations and a second subset of model components which are independent of directional lighting variations. A fit of the model to the face or certain other object is obtained including adjusting one or more individual values of one or more of the model components of the linear texture model. Based on the obtained fit of the model to the face or certain other object in the scene, a characteristic of the face or certain other object is determined.

Claims

1. A method of determining a characteristic of a face or certain other object within a scene captured in a digital image, comprising:

(a) acquiring a digital image including a face or certain other object within a scene;

(b) applying a linear texture model that is constructed based on a training data set and that comprises a class of objects including a first subset of model components that exhibit a dependency on directional lighting variations and a second subset of model components which are independent of directional lighting variations but which are dependent on at least one other characteristic of said face or certain other object;

(c) determining an initial location of the face or certain other object in the scene;

(d) adjusting values of the subset of model components that exhibit a dependency on directional lighting variations and at least one other of the model components of said linear texture model to obtain a desired fit between the model and said face or certain other object;

(e) based on the value of said at least one other model component, determining at least one characteristic of the face or certain other object; and

(f) displaying, transmitting, storing, applying face or other object recognition to, or editing, a corrected or otherwise processed version of the digital image, a value corresponding to the at least one characteristic or other data generated based on the determining, or combinations thereof.

2. The method of claim 1, further comprising:

(g) adjusting said characteristic of the face or certain other object by changing the fitted value of at least one model component to obtain an adjusted model; and

(h) correcting the appearance of said face or certain other object within the acquired scene by overlaying said adjusted model over the original face or certain other object and.

3. The method of claim 2, wherein said at least characteristic of the face or certain other object comprises sharpness, graininess, global luminance, color balance, skin tone, facial or object pose, a state of facial expression or a state of object appearance or combinations thereof, and wherein said adjusting is performed according to a pre-determined criterion, a set of programmed rules, a user adjustment, a criterion determined from the acquisition parameters of the original image, or a criterion determined from metadata associated with the original image or combinations thereof.

4. The method of claim 2, wherein said characteristic of the face or certain other object comprises a blemish or defect of the face or certain other object region, and the training set includes regions which are free of said defect, and the adjusting is performed so as to eliminate said defect.

5. The method of claim 4, wherein the blemish or defect is one of an acne blemish or abnormal skin texture;

a noise component;

a flash-eye defect;

an eye-blink;

a periodic variation in object luminance;

a motion blur, a focus blur, an underexposure of the region, an overexposure of the region, an unnatural color distribution across the region, an unnatural local contrast distribution across the region or combinations thereof.

6. The method of claim 5, wherein the blemish comprises an acne blemish or other skin blemish.

7. The method of claim 5, wherein the blemish comprises a photographic artefact.

8. The method of claim 1, wherein the linear texture model is constructed from two independently trained linear texture models the first model employing a set of training image chosen to emphasize directional lighting variations only;

the second model employing a set of training image chosen to emphasize at least one additional characteristic of the face or certain other object.

9. The method of claim 8, further comprising:

(i) generating a first set of training images from a master set of training images by subtracting more than one face region, subject to frontal illumination, from a corresponding set of face regions where each face region in the set has a different directional lighting characteristic; and

(ii) generating a second set of training images that comprise frontally illuminated faces.

10. The method of claim 1, wherein the model components comprise eigenvectors, and the individual values comprises eigenvalues of the eigenvectors.

11. The method of claim 1, wherein the at least one determined characteristic comprises a feature that is independent of directional lighting.

12. The method of claim 1, further comprising generating a reconstructed image without a periodic noise component, including obtaining a second fit of the face or certain other object to a second linear texture model that is based on a training data set and that comprises a class of objects including a set of model components which are without a periodic noise component.

13. The method of claim 12, further comprising:

(i) extracting the periodic noise component including determining a difference between the face or certain other object and the reconstructed image, and

(ii) determining a frequency of the noise component; and

(iii) removing the periodic noise component of the determined frequency.

14. The method of claim 1, further comprising determining an exposure value for the face or certain other object, including obtaining a fit to the face or certain other object to a second linear texture model that is based on a training data set and that comprises a class of objects including a set of model components that exhibit a dependency on exposure value variations.

15. The method of claim 14, further comprising including reducing an effect of a background region or density contrast caused by shadow or both.

16. The method of claim 1, further comprising controlling a flash to accurately reflect a lighting condition, including obtaining a flash control condition by referring to a reference table and controlling a flash light emission according to the flash control condition.

17. The method of claim 16, further comprising reducing an effect of contrasting density caused by shadow or black compression or white compression or combinations thereof

18. The method of claim 1, further comprising adjusting or determining a graininess value, or both, including:

(i) applying a second linear texture model that is constructed based on a training data set and comprises a class of objects including a subset of model components that exhibit a dependency on graininess variations,

(ii) obtaining a fit of said second model to said face or certain other object in the scene including adjusting one or more individual values of one or more model components of said second linear texture model; and

(iii) based on the obtained fit of the second model to said face or certain other object in the scene, adjusting a graininess of the face or certain other object including changing one or more values of one or more model components of the second linear texture model to generate a further adjusted object model.

19. The method of claim 1, further comprising converting, adjusting or determining a resolution value, or combinations thereof, including:

(i) applying a second linear texture model that is constructed based on a training data set and comprises a class of objects including a subset of model components that exhibit approximately a same resolution as said face or certain other object,

(ii) obtaining a fit of said second model to said face or certain other object in the scene including adjusting one or more individual values of one or more model components of said second linear texture model; and

(iii) based on the obtained fit of the second model to said face or certain other object in the scene, converting a resolution of the face or certain other object including changing one or more values of one or more model components of the second linear texture model to generate a further adjusted object model.

20. A digital image acquisition device including an optoelectonic system for acquiring a digital image including a processor, a lens and an image sensor, and a digital memory having stored therein processor-readable code for programming the processor to perform a method of determining a characteristic of a face or certain other object within a scene captured in a digital image, wherein the method comprises:

(a) acquiring a digital image including a face or certain other object within a scene;

(b) applying a linear texture model that is constructed based on a training data set and that comprises a class of objects including a first subset of model components that exhibit a dependency on directional lighting variations and a second subset of model components which are independent of directional lighting variations but which are dependent on at least one other characteristic of said face or certain other object;

(c) determining an initial location of the face or certain other object in the scene;

(d) adjusting values of the subset of model components that exhibit a dependency on directional lighting variations and at least one other of the model components of said linear texture model to obtain a desired fit between the model and said face or certain other object; and

(e) based on the value of said at least one other model component, determining at least one characteristic of the face or certain other object.

21. The device of claim 20, wherein the method further comprises:

(g) adjusting said characteristic of the face or certain other object by changing the fitted value of at least one model component to obtain an adjusted model; and

(h) correcting the appearance of said face or certain other object within the acquired scene by overlaying said adjusted model over the original face or certain other object and.

22. The device of claim 21, wherein said at least characteristic of the face or certain other object comprises sharpness, graininess, global luminance, color balance, skin tone, facial or object pose, a state of facial expression or a state of object appearance or combinations thereof, and wherein said adjusting is performed according to a pre-determined criterion, a set of programmed rules, a user adjustment, a criterion determined from the acquisition parameters of the original image, or a criterion determined from metadata associated with the original image or combinations thereof.

23. The device of claim 21, wherein said characteristic of the face or certain other object comprises a blemish or defect of the face or certain other object region, and the training set includes regions which are free of said defect, and the adjusting is performed so as to eliminate said defect.

24. The device of claim 23, wherein the blemish or defect is one of an acne blemish or abnormal skin texture;

a noise component;

a flash-eye defect;

an eye-blink;

a periodic variation in object luminance;

a motion blur, a focus blur, an underexposure of the region, an overexposure of the region, an unnatural color distribution across the region, an unnatural local contrast distribution across the region or combinations thereof.

25. The device of claim 24, wherein the blemish comprises an acne blemish or other skin blemish.

26. The device of claim 24, wherein the blemish comprises a photographic artefact.

27. The device of claim 20, wherein the linear texture model is constructed from two independently trained linear texture models the first model employing a set of training image chosen to emphasize directional lighting variations only;

the second model employing a set of training image chosen to emphasize at least one additional characteristic of the face or certain other object.

28. The device of claim 27, wherein the method further comprises:

(i) generating a first set of training images from a master set of training images by subtracting more than one face region, subject to frontal illumination, from a corresponding set of face regions where each face region in the set has a different directional lighting characteristic; and

(ii) generating a second set of training images that comprise frontally illuminated faces.

29. The device of claim 20, wherein the model components comprise eigenvectors, and the individual values comprises eigenvalues of the eigenvectors.

30. The device of claim 20, wherein the at least one determined characteristic comprises a feature that is independent of directional lighting.

31. The device of claim 20, further comprising generating a reconstructed image without a periodic noise component, including obtaining a second fit of the face or certain other object to a second linear texture model that is based on a training data set and that comprises a class of objects including a set of model components which are without a periodic noise component.

32. The device of claim 31, wherein the method further comprises:

(i) extracting the periodic noise component including determining a difference between the face or certain other object and the reconstructed image, and

(ii) determining a frequency of the noise component; and

(iii) removing the periodic noise component of the determined frequency.

33. The device of claim 20, wherein the method further comprises determining an exposure value for the face or certain other object, including obtaining a fit to the face or certain other object to a second linear texture model that is based on a training data set and that comprises a class of objects including a set of model components that exhibit a dependency on exposure value variations.

34. The device of claim 33, wherein the method further comprises reducing an effect of a background region or density contrast caused by shadow or both.

35. The device of claim 20, wherein the method further comprises controlling a flash to accurately reflect a lighting condition, including obtaining a flash control condition by referring to a reference table and controlling a flash light emission according to the flash control condition.

36. The device of claim 35, wherein the method further comprises reducing an effect of contrasting density caused by shadow or black compression or white compression or combinations thereof

37. The device of claim 20, wherein the method further comprises adjusting or determining a graininess value, or both, including:

(i) applying a second linear texture model that is constructed based on a training data set and comprises a class of objects including a subset of model components that exhibit a dependency on graininess variations,

(ii) obtaining a fit of said second model to said face or certain other object in the scene including adjusting one or more individual values of one or more model components of said second linear texture model; and

(iii) based on the obtained fit of the second model to said face or certain other object in the scene, adjusting a graininess of the face or certain other object including changing one or more values of one or more model components of the second linear texture model to generate a further adjusted object model.

38. The device of claim 20, wherein the method further comprises converting, adjusting or determining a resolution value, or combinations thereof, including:

(i) applying a second linear texture model that is constructed based on a training data set and comprises a class of objects including a subset of model components that exhibit approximately a same resolution as said face or certain other object,

(ii) obtaining a fit of said second model to said face or certain other object in the scene including adjusting one or more individual values of one or more model components of said second linear texture model; and

(iii) based on the obtained fit of the second model to said face or certain other object in the scene, converting a resolution of the face or certain other object including changing one or more values of one or more model components of the second linear texture model to generate a further adjusted object model.