FACE DETECTION AND TRACKING

• US 20130113956A1
• Filed: 11/04/2011
• Published: 05/09/2013
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Huitema, Christian F., Anderson, Sean Eron
• Granted Patent: US 9,148,637 B2
• Time in Patent Office: Days
• US Class Current: 348/223.1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/141 Control of illumination
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/22 by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 23/88 for colour balance, e.g. white-balance circuits or colour temperature control

Abstract

Methods and systems for face detection and tracking using an image-based capture device are disclosed herein. The method includes generating a depth image of a scene, generating a mask image from the depth image, and detecting a position of a face of a user in the scene using the mask image. The method also includes determining an intensity of the face using a first color channel of the mask image and adjusting a gain level of a first color channel of the image-based capture device directed at the scene to achieve a target intensity of the face.

Claims

1. A method for face detection and tracking using an image-based capture device, comprising:

generating a depth image of a scene;

generating a mask image from the depth image;

detecting a position of a face of a user in the scene using the mask image;

determining an intensity of the face using a first color channel of the mask image; and

adjusting a gain level of a first color channel of the image-based capture device directed at the scene to achieve a target intensity of the face.

2. The method of claim 1, comprising adjusting at least a second color channel gain level of the image-based capture device to achieve a color balance of the face based on the target intensity.

3. The method of claim 1, comprising adjusting an exposure time of the image-based capture device to improve a light level of the face.

4. The method of claim 3, comprising estimating the light level of the face based on a green gain level, a blue gain level, a red gain level, and an exposure time of the image-based capture device.

5. The method of claim 4, comprising increasing the exposure time of one or more cameras within the image-based capture device if the light level of the face is too low.

6. The method of claim 4, comprising providing feedback to the user if the light level of the face is above a first threshold value or below a second threshold value.

7. The method of claim 1, comprising:

detecting a position of a face for each of a plurality of users in the scene using the mask image;

determining an average of face intensities for the plurality of users using a first color channel of the mask image; and

adjusting a gain level of a first color channel of the image-based capture device directed at the scene to achieve a target face intensity for the plurality of users.

8. The method of claim 1, wherein detecting the position of the face of the user in the scene using the mask image comprises locating a proximate region of the face of the user versus a background within the mask image.

9. The method of claim 1, wherein determining the intensity of the face using the first color channel of the mask image comprises undoing a gamma correction of the mask image and averaging a mean intensity and a median intensity of the face within the mask image, wherein the mean intensity and the median intensity comprise intensities within the first color channel.

10. A face detection and tracking system, comprising:

an image-based capture device, comprising a camera component that generates a depth image and a color image of a scene; and

a computing device in operative communication with the image-based capture device, wherein the computing device comprises a processor configured to;

generate a mask image from the depth image and the color image;

detect a position of a face of a user in the scene using the mask image;

determine an intensity of the face using a green color channel of the mask image; and

direct the image-based capture device to adjust a green gain level of the green color channel to achieve a target intensity of the face.

11. The face detection and tracking system of claim 10, wherein the computing device comprises a processor configured to direct the image-based capture device to adjust an exposure time to improve a light level of the face.

12. The face detection and tracking system of claim 11, wherein the computing device comprises a processor configured to direct the image-based capture device to increase an exposure time if a filtered green gain level of the green color channel exceeds a high water mark threshold.

13. The face detection and tracking system of claim 11, wherein the computing device comprises a processor configured to direct the image-based capture device to decrease an exposure time if a filtered green gain level of the green color channel is below a low water mark threshold.

14. The face detection and tracking system of claim 10, wherein the computing device comprises a processor configured to direct the image-based capture device to adjust a red gain level or a blue gain level, or both, to achieve a color balance of the face.

15. The face detection and tracking system of claim 10, wherein the computing device comprises a processor configured to direct the image-based capture device to adjust a red gain level and a blue gain level to a fixed difference from the green gain level to achieve a basic skin color balance of the face.

16. The face detection and tracking system of claim 10, wherein the image-based capture device comprises an IR projector, an IR camera, and an RGB camera, and wherein the IR projector, the IR camera, and the RGB camera are temporally synchronized.

17. One or more non-volatile computer-readable storage media for storing computer readable instructions, the computer-readable instructions providing a face detection and tracking system when executed by one or more processing devices, the computer-readable instructions comprising code configured to:

detect a position of a face of a user within a capture area using a face detection module;

adjust a gain level of a first color channel of an image-based capture device using a settings adjustment module to achieve a target intensity of the face; and

adjust at least a second color channel gain level of the image-based capture device using the settings adjustment module to achieve a color balance of the face based on the target intensity.

18. The computer-readable storage media of claim 17, wherein the computer-readable instruction comprise code configured to:

adjust an exposure time of the image-based capture device using the settings adjustment module to improve a light level of the face.

19. The computer-readable storage media of claim 17, wherein the computer-readable instruction comprise code configured to:

track movements and features of the face using a face tracking module.

20. The computer-readable storage media of claim 17, wherein the gain level of the first color channel comprises a green gain level of a green color channel, and wherein at least the second color channel gain level comprises a red gain level of a red color channel or a blue gain level of a blue color channel, or both.