Skin Tone and Feature Detection for Video Conferencing Compression

• US 20110249756A1
• Filed: 04/07/2010
• Published: 10/13/2011
• Est. Priority Date: 04/07/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank
• Granted Patent: US 8,588,309 B2
• Time in Patent Office: Days
• US Class Current: 375/240.24
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

In many videoconferencing applications, bandwidth is at a premium, and thus, it is important to encode a given video frame intelligently. It is often desirable that a larger amount of information be spent encoding the more important parts of the video frame, e.g., human facial features, whereas the less important parts of the video frame can be compressed at higher rates. Thus, there is need for an apparatus, computer readable medium, processor, and method for intelligent skin tone and facial feature aware videoconferencing compression that can “suggest” intelligent macroblock compression ratios to a video encoder. The suggestion of compression rates can be based at least in part on a determination of which macroblocks in a given video frame are likely to contain skin tones, likely to contain features (e.g., edges), likely to contain features in or near skin tone regions, or likely to contain neither skin tones nor features.

Claims

1. A processor programmed to perform a video compression method, the method comprising:

receiving video data at the processor, the video data having a frame with a plurality of regions, each of the regions having a plurality of pixels;

determining with the processor any first pixels in each region having a predetermined tone;

determining with the processor any second pixels in each region that are part of at least one feature;

scoring with the processor each of the regions of the video frame based on the determination of any first and second pixels in the region; and

compressing with the processor the regions of the video frame based on the scoring.

2. The processor of claim 1, wherein determining the first pixels comprises comparing first values of the pixels in the regions to the predetermined tone.

3. The processor of claim 2, wherein the predetermined tone comprises CbCr values indicative of human skin tones.

4. The processor of claim 2, wherein the predetermined tone comprises RGB values indicative of human skin tones.

5. The processor of claim 1, wherein determining the second pixels comprises carrying out an edge detection process on the pixels in the regions.

6. The processor of claim 1, wherein determining the second pixels comprises carrying out a feature detection process on the pixels in the regions.

7. The processor of claim 1, wherein scoring each of the regions comprises assigning a given pixel of each region with a first score if the given pixel has the predetermined tone.

8. The processor of claim 7, wherein scoring each of the regions comprises assigning a given pixel of each region with a second score if the given pixel is part of at least one feature.

9. The processor of claim 8, wherein the second score is greater than the first score.

10. The processor of claim 8, wherein scoring each of the regions comprises assigning a given pixel of each region with a third score if the given pixel is part of at least one feature and within a threshold distance of a pixel that has the predetermined tone.

11. The processor of claim 10, wherein the third score is greater than the second score, and wherein the second score is greater than the first score.

12. The processor of claim 10, wherein scoring each of the regions comprises averaging values assigned to the pixels in each region and associating the average value to the region.

13. The processor of claim 1, wherein compressing the regions comprises compressing the regions having a higher score with less compression than the regions having a lower score.

14. The processor of claim 1, wherein compressing the regions comprises:

comparing the score of a given one of the regions to the scores of one or more neighboring regions; and

adjusting the score of the given region based on a discrepancy.

15. The processor of claim 14, wherein the discrepancy is indicative of the given region having a lower or higher score compared to the one or more neighboring regions.

16. The processor of claim 1, wherein each of the regions comprises a macroblock.

25. A computer usable medium having a computer readable program code embodied therein, wherein the computer readable program code is adapted to be executed to implement the method performed by the programmed processor of claim 1.

17. A video compression method, comprising:

receiving video data, the video data having a frame with a plurality of regions, each of the regions having a plurality of pixels;

determining any first pixels in each region having a color value within a predetermined tone region;

determining any second pixels in each region that are part of at least one feature;

for each pixel in each region;

assigning the pixel a first value if the pixel is a first pixel but not a second pixel;

assigning the pixel a second value if the pixel is a second pixel but not a first pixel;

assigning the pixel a third value if the pixel is neither a first pixel nor a second pixel; and

assigning the pixel a fourth value if the pixel is both a second pixel and either a first pixel or within a threshold distance of a first pixel;

scoring each of the regions of the video frame based on the average assigned value of the pixels in the region; and

compressing each of the regions of the video frame based on the region'"'"'s score.

18. The method of claim 17, wherein the fourth value is larger than the first, second, and third values.

19. The method of claim 18, wherein the first and second values are each larger than the third value.

20. A video compression method, comprising:

receiving video data, the video data having a frame with a plurality of regions, each of the regions having a plurality of pixels;

determining any first pixels in each region having a color value within a predetermined tone region;

determining any second pixels in each region that are part of at least one feature;

scoring each of the regions based on the determination of any first and second pixels in the region;

correcting the score for each region having a score that differs from the score of one or more neighboring regions by more than a threshold value to be equal to the average score of the neighboring regions; and

compressing each of the regions of the video frame based on the region'"'"'s score.

21. The method of claim 20, wherein the predetermined tone region comprises CbCr values indicative of human skin tones.

22. The method of claim 20, wherein determining the second pixels comprises carrying out a feature detection process on the pixels in the regions.

23. An apparatus, comprising:

an image sensor for obtaining video data;

memory operatively coupled to the image sensor; and

a processor operatively coupled to the memory and the image sensor and programmed to encode the video data, the processor configured to;

receive video data, the video data having a frame with a plurality of regions, each of the regions having a plurality of pixels;

determine any first pixels in each region having a predetermined tone;

determine any second pixels in each region that are part of at least one feature;

score each of the regions of the video frame based at least in part on the determination of any first and second pixels in the region; and

compress the regions of the video frame based at least in part on the scoring.

24. The apparatus of claim 23, wherein the apparatus comprises at least one of the following:

a digital camera, digital video camera, mobile phone, personal data assistant, portable music player, and computer.