Automatic skin color model face detection and mean-shift face tracking

• US 7,869,631 B2
• Filed: 12/11/2006
• Issued: 01/11/2011
• Est. Priority Date: 12/11/2006
• Status: Active Grant

• Current Assignee: ArcSoft, Inc.
• Original Assignee: ArcSoft, Inc.
• Inventors: Xu, Xiao-Dong, Wang, Jin
• Primary Examiner(s): Werner; Brian P
• Assistant Examiner(s): DRENNAN, BARRY T
• Application Number:
  US11/636,487
• Publication Number:
  US 20080137957A1
• Time in Patent Office: 1,492 Days
• Field of Search: 382/103, 382/118
• US Class Current: 382/118
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

The present invention discloses a method of automatically detecting and tracking a face by an electronic capturing device that alternatively uses a face detecting algorithm to quickly locate a face in a frame based on a skin color model and a face tracking algorithm to locate a face in subsequent frames by a nonparametric technique and a mean shift algorithm. If the face tracking algorithm cannot track and locate a face correctly, the face detecting algorithm will be used again to detect a face position in another new frame until the face position is located successfully, and then the face tracking algorithm will be used again for detecting and locating the face position in subsequent frames. After this method has detected a face position, a variable focal lens is used to slowly and smoothly refocus the frame including a face region, so as to obtain a clear face image.

Claims

1. A method of automatically detecting and tracking a face by an electronic capturing device, comprising:

executing a face detecting algorithm to detect a face position in a frame in a red, green and blue (RGB) chromaticity space based on a skin color model, wherein said face detecting algorithm performs a color transformation for transforming said frame into a new frame in a hue, saturation and value (HSV) chromaticity space and comprises steps of;

calculating a histogram of hue of color pixels on said new frame when r>

max(g,b, S) and S>

38, wherein S represents saturation of said new frame, and r>

max(g,b,S) and S>

38 represents that intensity of red color of said frame is larger than maximum value of intensities of green color and blue color of said frame when the saturation S of said new frame is larger than 38;

obtaining a maximum value Maxh of said histogram of hue when number of said color pixels multiplied by 256 is larger than total number of all pixels on said new frame;

obtaining a smallest value i of said histogram of hue, such that said pixels of said new frame with hues falling within a histogram hue range [Maxh−

i, Maxh+i] are determined to be skin color pixels respectively;

forming a mask for said skin color pixels;

marking a label for all continuous regions in said new frame for defining a rectangle corresponding to said label when a number of pixels of said continuous regions is larger than the total number of pixels on said new frame divided by 256;

using a face feature matching engine to analyze and determine whether an image within said rectangle is a face or not;

generating a detected rectangle for covering at least one rectangle;

calculating a skin color probability density function for an image within said detected rectangle; and

outputting information of detecting a face within said detected rectangle and displaying said detected rectangle on said new frame corresponding to said face; and

executing a face tracking algorithm to detect face positions in subsequent new frames by using a nonparametric technique and a mean shift algorithm according to said face.

2. The method of claim 1, wherein said face detecting algorithm determines that color of a pixel x in said new frame belongs to a face skin color when l≦

h(x)≦

u, wherein h(x) represents the hue of pixel x, and u and l represent upper and lower thresholds of the hue of said face skin color respectively and said thresholds are used for obtaining a binary face image.

3. The method of claim 2, wherein said face tracking algorithm tracks a new position and a new scalar of a face in a subsequent new frame next to said new frame according to said face color probability distribution obtained by said face detecting algorithm from said new frame.

4. The method of claim 3 wherein, when a back projection B(i, j) of said subsequent new frame defining a skin color probability P(F(i, j)) of pixels of said subsequent new frame corresponding to pixels F(i, j) of said face in said new frame is obtained, said mean shift algorithm is used for locating a local peak mode of a said face in said subsequent frame close to said face in said new frame on said back projection B(i, j).

5. The method of claim 4, wherein said face tracking algorithm comprises:

reading said skin color probability density function of said face within said detected rectangle on said new frame;

computing said back projection of said subsequent new frame corresponding to said face in said new frame;

using said mean shift algorithm to perform an iteration for defining a new detected rectangle on said subsequent new frame;

computing pixels of an image within said new detected rectangle;

computing a scalar of said image within said new detected rectangle when said iteration is completed;

computing a mean zero moment of said back projection, and said zero moment is total sum of said back projection, and said mean zero moment is provided for representing a mean probability of colors of pixels in said new detected rectangle belonging to a skin color, and when said mean probability is greater than a predetermined threshold, then said image in said new detected rectangle is considered to be said face; and

outputting information of detecting said face within said new detected rectangle and displaying said new detected rectangle on said subsequent new frame corresponding to said face when said mean zero moment is greater than a predetermined threshold.

6. The method of claim 5, further comprising:

driving a variable focal lens of said electronic capturing device to refocus said face in said subsequent new frame, when said face in said subsequent new frame is detected.

7. The method of claim 6, wherein when said new detected rectangle R_k is confirmed, at least one transitional frame is added between said new detected rectangle R_k and a showing window RS_k-1 of said electronic capturing device by performing steps of:

setting RS₀=R₀ for a first new frame;

calculating a x-axis direction shift of said new detected rectangle according to an equation {right arrow over (x)}=center_Rx·

x−

center_RSk-1·

x, where center_Rk and center_RSk-1 represent centers of said new detected rectangle R_k and said showing window RS_k-1 repectively;

calculating a y-axis direction shift of said new detected rectangle according to an equation {right arrow over (y)}=center_Rk·

−

center_Rsk-1·

y;

calculating said scale of said new detected rectangle according to an equation

8. The method of claim 7, when refocusing said face within said new detected rectangle R_k, further comprising steps of:

shifting said new detected rectangle R_k in a x-axis direction same as said detected rectangle, when sumx>

8 and

9. The method of claim 2 wherein, when obtaining a plurality of detected rectangles, said face feature matching engine determines that images within said detected rectangles are candidates, matches each of said candidates with an ellipse to determine whether or not said candidate contains a face, and compares a ratio of long radius to short radius of said ellipse corresponding to said candidate with a predetermined threshold;

when the ratio of long radius to short radius of said ellipse corresponding to said candidate is greater than said predetermined threshold, then a face region corresponding to said candidate is discarded.

10. The method of claim 9, wherein said face tracking algorithm tracks a new position and a new scalar of a face in a subsequent new frame next to said new frame according to said face color probability distribution obtained by said face detecting algorithm from said new frame.

11. The method of claim 10 wherein, when a back projection B(i, j) of said subsequent new frame defining a skin color probability P(F(i, j)) of pixels of said subsequent new frame corresponding to pixels F(i, j) of said face in said new frame is obtained, said mean shift algorithm is used for locating a local peak mode of said face in said subsequent frame close to said face in said new frame on said back projection B(i, j).

12. The method of claim 11, wherein said face tracking algorithm comprises:

reading said skin color probability density function of said face within said detected rectangle on said new frame;

computing said back projection of said subsequent new frame corresponding to said face in said new frame;

using said mean shift algorithm to perform an iteration for defining a new detected rectangle on said subsequent new frame;

computing pixels of an image within said new detected rectangle;

computing a scalar of said image within said new detected rectangle when said iteration is completed;

computing a mean zero moment of said back projection, and said zero moment is total sum of said back projection, and said mean zero moment is provided for representing a mean probability of colors of pixels in said new detected rectangle belonging to a skin color, and when said mean probability is greater than a predetermined threshold, then said image in said new detected rectangle is considered to be said face; and

outputting information of detecting said face within said new detected rectangle and displaying said new detected rectangle on said subsequent new frame corresponding to said face when said mean zero moment is greater than a predetermined threshold.

13. The method of claim 12, further comprising:

driving a variable focal lens of said electronic capturing device to refocus said face in said subsequent new frame, when said face in said subsequent new frame is detected.

14. The method of claim 13, wherein when said new detected rectangle R_k is confirmed, at least one transitional frame is added between said new detected rectangle R_k and a showing window RS_k-1 of said electronic capturing device by performing steps of:

setting RS₀=R₀ for a first new frame;

calculating a x-axis direction shift of said new detected rectangle according to an equation {right arrow over (x)}=center_Rk·

x−

center_RSk-1·

x, where center R_k and center RS_k-1 represent centers of said new detected rectangle R_k and said showing window RS_k-1 repectively;

calculating a y-axis direction shift of said new detected rectangle according to an equation {right arrow over (y)}=center_Rk·

y−

center_RSk-1·

y;

calculating said scale of said new detected rectangle according to an equation

15. The method of claim 14, when refocusing said face within said new detected rectangle R_k, further comprising steps of:

shifting said new detected rectangle R_k in a x-axis direction same as said detected rectangle, when sumx>

8 and

16. The method of claim 9, wherein said candidate is matched with said ellipse by applying a principal components analysis (PCA) technique on a pixel x_i on said new frame, where x_i is in form of a column vector and i=1, . . . , n, and using a sample mean vector ũ

and a sample covariance matrix {tilde over (Σ

)} to estimate following mean vector μ and

covariance matrix Σ

;

17. The method of claim 16, wherein said face tracking algorithm tracks a new position and a new scalar of a face in a subsequent new frame next to said new frame according to said face color probability distribution obtained by said face detecting algorithm from said new frame.

18. The method of claim 17 wherein, when a back projection B(i, j) of said subsequent new frame defining a skin color probability P(F(i, j)) of pixels of said subsequent new frame corresponding to pixels F(i, j) of said face in said new frame is obtained, said mean shift algorithm is used for locating a local peak mode of a said face in said subsequent frame close to said face in said new frame on said back projection B(i, j).

19. The method of claim 18, wherein said face tracking algorithm comprises:

reading said skin color probability density function of said face within said detected rectangle on said new frame;

computing said back projection of said subsequent new frame corresponding to said face in said new frame;

using said mean shift algorithm to perform an iteration for defining a new detected rectangle on said subsequent new frame;

computing pixels of an image within said new detected rectangle;

computing a scalar of said image within said new detected rectangle when said iteration is completed;

computing a mean zero moment of said back projection, and said zero moment is total sum of said back projection, and said mean zero moment is provided for representing a mean probability of colors of pixels in said new detected rectangle belonging to a skin color, and when said mean probability is greater than a predetermined threshold, then said image in said new detected rectangle is considered to be said face; and

outputting information of detecting said face within said new detected rectangle and displaying said new detected rectangle on said subsequent new frame corresponding to said face when said mean zero moment is greater than a predetermined threshold.

20. The method of claim 19, further comprising:

driving a variable focal lens of said electronic capturing device to refocus said face in said subsequent new frame, when said face in said subsequent new frame is detected.

21. The method of claim 20, wherein when said new detected rectangle R_k is confirmed, at least one transitional frame is added between said new detected rectangle R_k and a showing window RS_k-1 of said electronic capturing device by performing steps of:

setting RS₀=R₀ for a first new frame;

calculating a x-axis direction shift of said new detected rectangle according to an equation {right arrow over (x)}=center_Rk·

x−

center_RSk-1·

x, where center R_k and center_RSk-1 represent centers of said new detected rectangle R_k and said showing window RS_k-1 repectively;

calculating a y-axis direction shift of said new detected rectangle according to an equation {right arrow over (y)}=center_Rk·

y−

center_RSk-1·

y;

calculating said scale of said new detected rectangle according to an equation

22. The method of claim 21, when refocusing said face within said new detected rectangle R_k, further comprising steps of:

shifting said new detected rectangle R_k in a x-axis direction same as said detected rectangle, when sumx >

8 and

23. The method of claim 16, wherein center of said ellipse is a mean vector, rotation angle of said ellipse is a direction of said main eigenvector with a major axis and a minor axis selected respectively to be r×

√

{square root over (λ

₁)} and r×

√

{square root over (λ

₂)}, r is a ratio falling in a range of numeric values from 1.8 to 2.2, a discard formula

24. The method of claim 23, wherein said face tracking algorithm tracks a new position and a new scalar of a face in a subsequent new frame next to said new frame according to said face color probability distribution obtained by said face detecting algorithm from said new frame.

25. The method of claim 24 wherein, when a back projection B(i, j) of said subsequent new frame defining a skin color probability P(F(i, j)) of pixels of said subsequent new frame corresponding to pixels F(i, j) of said face in said new frame is obtained, said mean shift algorithm is used for locating a local peak mode of a said face in said subsequent frame close to said face in said new frame on said back projection B(i, j).

26. The method of claim 25, wherein said face tracking algorithm comprises:

reading said skin color probability density function of said face within said detected rectangle on said new frame;

computing said back projection of said subsequent new frame corresponding to said face in said new frame;

using said mean shift algorithm to perform an iteration for defining a new detected rectangle on said subsequent new frame;

computing pixels of an image within said new detected rectangle;

computing a scalar of said image within said new detected rectangle when said iteration is completed;

computing a mean zero moment of said back projection, and said zero moment is total sum of said back projection, and said mean zero moment is provided for representing a mean probability of colors of pixels in said new detected rectangle belonging to a skin color, and when said mean probability is greater than a predetermined threshold, then said image in said new detected rectangle is considered to be said face; and

outputting information of detecting said face within said new detected rectangle and displaying said new detected rectangle on said subsequent new frame corresponding to said face when said mean zero moment is greater than a predetermined threshold.

27. The method of claim 26, further comprising:

driving a variable focal lens of said electronic capturing device to refocus said face in said subsequent new frame, when said face in said subsequent new frame is detected.

28. The method of claim 27, wherein when said new detected rectangle R_k is confirmed, at least one transitional frame is added between said new detected rectangle R_k and a showing window RS_k-1 of said electronic capturing device by performing steps of:

setting RS₀=R₀ for a first new frame;

calculating a x-axis direction shift of said new detected rectangle according to an equation {right arrow over (x)}=center_Rx·

x−

center_RSk-1·

x, where center_Rk and center_RSk-1 represent centers of said new detected rectangle R_k and said showing window RS_k-1 repectively;

calculating a y-axis direction shift of said new detected rectangle according to an equation {right arrow over (y)}=center_Rk·

y−

center_RSk-1·

y;

calculating said scale of said new detected rectangle according to an equation

29. The method of claim 28, when refocusing said face within said new detected rectangle R_k, further comprising steps of:

shifting said new detected rectangle R_k in a x-axis direction same as said detected rectangle, when sumx>

8 and