Facial image processing methods and systems

• US 7,492,927 B2
• Filed: 05/14/2004
• Issued: 02/17/2009
• Est. Priority Date: 08/30/2000
• Status: Expired due to Term

• Current Assignee: Zhigu Holdings Limited
• Original Assignee: Microsoft Corporation
• Inventors: Marschner, Stephen, Kang, Sing Bing, Olynyk, Kirk, Guenter, Brian K., Raghupathy, Sashi
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): RAHMJOO, MANUCHER
• Application Number:
  US10/846,308
• Publication Number:
  US 20040210427A1
• Time in Patent Office: 1,740 Days
• Field of Search: 382/154, 382/224, 382/118, 345/420, 345/426
• US Class Current: 382/118
• CPC Class Codes: G06T 15/506 Illumination models
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/60 relating to illumination properties, e.g. using a reflectance or lighting model
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

In the described embodiment, methods and systems for processing facial image data for use in animation are described. In one embodiment, a system is provided that illuminates a face with illumination that is sufficient to enable the simultaneous capture of both structure data, e.g. a range or depth map, and reflectance properties, e.g. the diffuse reflectance of a subject'"'"'s face. This captured information can then be used for various facial animation operations, among which are included expression recognition and expression transformation.

Claims

1. A facial image-processing method comprising:

illuminating a face with illumination, wherein the illumination comprises;

a first light source that is polarized and selected so that specularly-suppressed reflective properties of the face can be ascertained; and

a second light source that projects a structured pattern onto the face, while simultaneously illuminating the face with the first light source; and

contemporaneously capturing structure data describing the face'"'"'s structure and reflectance data describing reflectance properties of the face from the illumination, whereby the captured structure data and reflectance data is communicated to a computer to be processed.

2. The method of claim 1, wherein said illuminating comprises using additional light sources.

3. The method of claim 2, wherein one of the additional light sources comprises an infrared light source.

4. The method of claim 2, wherein one of the additional light sources comprises a narrow-band light source.

5. The method of claim 1, wherein the second light source projects the structured pattern on the face from which the structure data can be ascertained.

6. The method of claim 1, wherein the illuminating further comprises an infrared light source.

7. The method of claim 1, wherein the first and second light sources comprise infrared light sources.

8. The method of claim 1, wherein said illuminating comprises using multiple polarized light sources.

9. The method of claim 1, wherein said illuminating comprises illuminating the face with light sources at different frequencies.

10. The method of claim 1, wherein said capturing comprises using a camera having a polarizer that suppresses specularly-reflected light so that diffuse component reflection data is captured.

11. The method of claim 10, wherein the second light source projects the structured pattern on the face from which the structure data can be ascertained.

12. The method of claim 11, wherein the second light source comprises an infrared light source.

13. The method of claim 1, wherein the first and second light sources are narrow-band light sources.

14. A facial image-processing method comprising:

illuminating a face with a first light source that is polarized and selected so that specularly-suppressed reflective properties of the face can be ascertained;

illuminating the face with a second light source that projects a structured pattern onto the face, while simultaneously illuminating the face with the first light source;

capturing both specularly-suppressed reflection data and structure data from the simultaneous illumination, whereby the captured specularly-suppressed reflection data and structure data is communicated to a computer to be processed.

15. The method of claim 14, wherein the first and second light sources provide light at different frequencies.

16. The method of claim 14, wherein the first and second light sources provide infrared light.

17. The method of claim 14 further comprising processing the captured data to provide both (a) data that describes dimensional aspects of the face and (b) data that describes diffuse reflective properties of the face.

18. The method of claim 17, wherein the data that describes the diffuse reflective properties of the face comprises an albedo map.

19. A facial image processing system comprising:

a facial illumination system that is configured to provide multiple different light sources at the same time for illuminating a subject'"'"'s face, the multiple different light sources comprising;

a first light source that is polarized and selected so that specularly-suppressed reflective properties of the face can be ascertained; and

a second light source that projects a structured pattern onto the face, while simultaneously illuminating the face with the first light source; and

a data-capturing system configured to capture both structure data and reflectance data from the subject'"'"'s face when illuminated by the facial illumination system, whereby the captured structure data and reflectance data is communicated to a computer to be processed.

20. The system of claim 19, wherein the illumination system comprises an additional polarized light source.

21. The system of claim 19, wherein the illumination system comprises multiple polarized light sources.

22. The system of claim 19, wherein the second light source comprises an infrared light source.

23. The system of claim 19, wherein the different light sources are all infrared light sources.

24. The system of claim 19, wherein at least one of the different light sources comprises an infrared light source.

25. The system of claim 19, wherein the different light sources are selected to comprise narrow-band light sources.