Video image quality

• US 7,542,600 B2
• Filed: 10/21/2004
• Issued: 06/02/2009
• Est. Priority Date: 10/21/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Jiang, Yu, Keman, Li, Shipeng
• Primary Examiner(s): Do; Anh Hong
• Application Number:
  US10/970,263
• Publication Number:
  US 20060088209A1
• Time in Patent Office: 1,685 Days
• Field of Search: 382/167, 382/164, 382/166, 382/282, 382/274, 382/103, 382/107, 382/118, 382/199, 382/159, 382/173, 348/222.1, 348/207.99, 348/231.99, 348/317, 348/14.15, 348/169, 348/597, 358/1.9, 358/537, 358/515, 358/520, 707/6
• US Class Current: 382/167
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Video image quality may be improved by correcting exposure levels and/or enhancing contrast amounts on each frame. One or more of the following phases may be implemented: skin-color model building, face detecting, exposure level correcting, and contrast enhancing. In a described implementation, a Gaussian skin-color model is built for each image frame during runtime. The Gaussian skin-color model is built with training pixels that are selected responsive to a defined skin color range, which is created offline from manually-selected skin pixels of multiple test sequences. In another described implementation, each pixel of an image frame is re-exposed using a ratio of contrast amount control variables (CACVs). More specifically, a pixel may be converted to a corresponding light intensity using a first CACV, and the corresponding light intensity may be reconverted to a pixel using a second CACV to enhance the contrast and possibly reduce fuzziness of the image frame.

Claims

1. One or more processor-accessible computer readable medium comprising processor-executable instructions that, when executed, direct a computing device to perform actions comprising:

selecting, via the computing device, training pixels from an image frame based on a defined skin color range;

building a skin-color model using the selected training pixels;

detecting a skin portion of the image frame using the skin-color model; and

adjusting the image frame with regard to the detected skin portion.

2. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the action of adjusting comprises an action of:

correcting an exposure level of the image frame responsive to an average luminance value of the detected skin portion.

3. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the action of adjusting comprises an action of:

enhancing a contrast amount of the image frame using an exposure-density function.

4. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the action of building comprises an action of:

building a two-dimensional Gaussian skin-color model.

5. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the action of selecting comprises an action of:

selecting the training pixels based on the defined skin color range;

(i) the defined skin color range comprising at least one range of Gaussian distribution centers, and (ii) the defined skin color range created offline using a collection of test sequences, respective sets of skin pixels manually selected from respective ones of the collected test sequences, and respective Gaussian distribution centers calculated from respective Gaussian skin-color models built responsive to the respective sets of manually-selected skin pixels of the respective ones of the collected test sequences.

6. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the action of selecting comprises an action of:

selecting the training pixels based on the defined skin color range in a YCbCr color space, the defined skin color range including a range corresponding to Cb and a range corresponding to Cr.

7. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the processor-executable instructions, when executed, direct the computing device to perform further actions comprising:

determining a center of the skin-color model; and

ascertaining if the determined center comports with at least one reliability range;

wherein if the determined center is ascertained to comport with the at least one reliability range, the action of detecting comprises using a current skin-color model built for a current image frame to detect the skin portion of the current image frame.

8. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the processor-executable instructions, when executed, direct the computing device to perform further actions comprising:

determining a center of the skin-color model; and

ascertaining if the determined center comports with at least one reliability range;

wherein if the determined center is not ascertained to comport with the at least one reliability range, the action of detecting comprises using a previous skin-color model built for a previous image frame to detect the skin portion of a current image frame.

9. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the processor-executable instructions, when executed, direct the computing device to perform further actions comprising:

determining a center of the skin-color model; and

ascertaining if the determined center comports with at least one reliability range;

wherein the at least one reliability range comprises a summed reliability range and a difference reliability range that are produced offline from tracked centers of multiple respective Gaussian skin-color models of multiple respective test sequences.

10. The one or more processor-accessible computer readable medium as recited in claim 1, wherein the one or more processor-accessible media comprise at least one of (i) one or more storage media or (ii) one or more transmission media.

11. A device, comprising:

one or more processors; and

memory having instructions executable by the one or more processors, the memory including;

a data collection module to accept a defined skin color range and an image frame as an input;

a selection module to select training pixels from the image frame based on the defined skin color range; and

a skin-color model builder module to build a skin-color model at runtime for the image frame using the selected training pixels.

12. The device as recited in claim 11, wherein the device is at least one of (i) a mobile telephone, (ii) video conferencing equipment, or (iii) at least part of a personal computer.

13. The device as recited in claim 11, wherein the defined skin color range is expressed in terms of chrominance-B and chrominance-R in a YCbCr color space.

14. The device as recited in claim 11, wherein the skin-color model builder module is further operable to scan the image frame for pixels having chrominance values fitting within the defined skin color range and to select those pixels as the training pixels.

15. The device as recited in claim 11, wherein the skin-color model builder is further operable to:

accept at least one reliability range as an input; and

perform a reliability examination on the skin-color model built for the image frame using the at least one reliability range.

16. The device as recited in claim 15, wherein the skin-color model built by the skin-color model builder module includes a Gaussian skin-color model;

and wherein the at least one reliability range includes a summed reliability range and a difference reliability range that are determined from multiple tracked centers of multiple Gaussian skin-color models from multiple test sequences, and wherein the reliability examination comprises ascertaining if a center of the built Gaussian skin-color model comports with the summed reliability range and the difference reliability range.

17. The device as recited in claim 16, further comprising:

a face detector that is adapted to detect at least part of a face of the image frame utilizing a given Gaussian skin-color model;

wherein the Gaussian skin-color model built by the skin-color model builder for the image frame is utilized by the face detector to detect the at least part of the face when the built Gaussian skin-color model is ascertained to comport with the summed reliability range and the difference reliability range, and another Gaussian skin-color model built for a previous image frame is utilized by the face detector to detect the at least part of the face when the built Gaussian skin-color model is ascertained to not comport with the summed reliability range and the difference reliability range.

18. The device as recited in claim 11, further comprising:

a face detector that is a capable of accepting the skin-color model from the skin-color model builder, the face detector adapted to scan pixels of the image frame and to use the skin-color model to add a particular pixel to a set of face pixels if a probability that the particular pixel belongs to a skin color class exceeds a predetermined skin pixel probability threshold.

19. The device as recited in claim 11, further comprising:

an exposure level corrector that is adapted (i) to calculate a difference between an ideal exposure level and an average luminance value for a set of face pixels that are detected in the image frame using the skin-color model and (ii) to re-expose each pixel from the image frame responsive to the calculated difference.

20. The device as recited in claim 11, further comprising:

a contrast enhancer that is adapted to convert each pixel from the image frame to a corresponding light intensity using a first contrast amount control variable value and to reconvert the corresponding light intensity back to a pixel value using a second contrast amount control variable value,

21. The device as recited in claim 11, further comprising:

an exposure level corrector and contrast enhancer that is adapted to re-expose each pixel from the image frame utilizing an exposure-density function (i) responsive to a difference between an ideal exposure level and an average luminance value of a set of skin pixels of the image frame and (ii) using a ratio of contrast amount control variables.

22. An arrangement for video image processing in conjunction with skin color modeling, the arrangement comprising:

selection means for selecting training pixels from an image frame using a predefined skin color range;

build means for building a skin-color model using the selected training pixels; and

detection means for detecting a skin portion of the image frame using the skin-color model.

23. The arrangement as recited in claim 22, further comprising:

means for conducting a reliability examination to ascertain if at least one attribute of the skin-color model comports with a reliability range; and

utilization means for utilizing the skin-color model built for the image frame when detecting a skin portion in the image frame if the at least one attribute is ascertained to comport with the reliability range and for utilizing another skin-color model built for a previous image frame when detecting a skin portion in the image frame if the at least one attribute is not ascertained to comport with the reliability range.

24. The arrangement as recited in claim 22, wherein the detection means for detecting a skin portion of the image frame further includes using a skin pixel probability threshold.

25. The arrangement as recited in claim 22, further comprising:

correction means for correcting an exposure level of the image frame responsive to an ideal exposure level and an average luminance value of a set of skin pixels detected using the skin-color model.

26. The arrangement as recited in claim 22, further comprising:

enhancement means for enhancing a contrast amount of the image frame by re-exposing pixels of the image frame using a ratio of contrast amount control variables.

27. The arrangement as recited in claim 22, further comprising:

combination means for correcting an exposure level and enhancing a contrast amount of the image frame using an exposure-density function along with (i) a difference between an ideal exposure level and an average luminance value of a skin portion detected using the skin-color model and (ii) a ratio of contrast amount control variables.

28. The arrangement as recited in claim 22, wherein the arrangement comprises at least one of (i) one or more processor-accessible media or (ii) at least one device.

29. A method, comprising:

building a skin-color model, via a computing device, for an image frame using a defined skin color range;

detecting a facial portion in the image frame using the skin-color model; and

correcting an exposure level of the image frame based on a luminance value of the detected facial portion.

30. The method as recited in claim 29, wherein the building the skin-color model includes building the skin-color model at runtime for multiple frames.

31. The method as recited in claim 30, wherein the building a skin-color model includes building the skin-color model for one image frame out of every predetermined number of image frames.

32. The method as recited in claim 30, wherein the skin-color model comprises a Gaussian skin-color model;

and wherein the defined skin color range is created offline using a collection of test sequences;

respective sets of skin pixels manually selected from respective ones of the collected test sequences, and respective Gaussian distribution centers calculated from respective Gaussian skin-color models built responsive to the respective sets of manually-selected skin pixels of the respective ones of the collected test sequences;

the defined skin color range created based on the calculated respective Gaussian distribution centers.

33. The method as recited in claim 29, further comprising enhancing a contrast amount of the image frame.

34. The method as recited in claim 29, wherein the luminance value comprises an average luminance value;

and

further comprising skipping the action of correcting when the average luminance value of the detected facial portion is relatively close to an ideal exposure level.

35. The method as recited in claim 29, wherein the detecting a facial portion in the image frame includes determining if respective individual probabilities associated with respective individual pixels of multiple pixels of the image frame exceed a predetermined threshold, each respective individual probability indicating if an associated respective individual pixel is likely to belong to a skin color class in accordance with the skin-color model.

36. The method as recited in claim 29, wherein the building a skin-color model includes:

selecting training pixels from the image frame based on the defined skin color range; and

building the skin-color model using the selected training pixels.

37. The method as recited in claim 29, wherein the skin-color model comprises a Gaussian skin-color model;

wherein the building a skin-color model includes;

selecting training pixels from the image frame based on the defined skin color range; and

building the Gaussian skin-color model using the selected training pixels; and

the method further comprising;

determining a center of the Gaussian skin-color model;

ascertaining if the determined center corn ports with at least one reliability range;

if the determined center is ascertained to comport with the at least one reliability range, utilizing the Gaussian skin-color model built for the image frame in the action of detecting; and

if the determined center is not ascertained to comport with the at least one reliability range, utilizing a previous Gaussian skin-color model built for a previous image frame in the action of detecting.

38. The method as recited in claim 37, wherein the at least one reliability range comprises a summed reliability range and a difference reliability range, the summed and difference reliability ranges produced from multiple tracked Gaussian centers of multiple Gaussian skin-color models built offline from multiple test sequences.

39. A runtime method comprising:

using a selecting module for selecting training pixels from an image frame based on a predefined skin color range, the predefined skin color range created offline;

using a skin color model builder module for building a Gaussian skin-color model using the selected training pixels;

using a detecting module for detecting a facial portion of the image frame using the Gaussian skin-color model and a skin pixel probability threshold; and

using a correcting module for correcting an exposure level of the image frame with regard to the detected facial portion.