Method and apparatus for facial image processing

• US 9,754,153 B2
• Filed: 11/21/2013
• Issued: 09/05/2017
• Est. Priority Date: 11/23/2012
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Li, Jiangwei, Wang, Kongqiao, Yan, He
• Primary Examiner(s): Cese, Kenny
• Application Number:
  US14/086,235
• Publication Number:
  US 20140147003A1
• Time in Patent Office: 1,384 Days
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The present invention provides a method for image processing, a corresponding apparatus and a computer program product. The method comprises performing face detection of an image, obtaining a coarse face segmentation region of at least one face and a contour edge of the at least one face based on the face detection; and adjusting the coarse face segmentation region based on the contour edge to obtain a fine face segmentation region. By using the method, corresponding apparatus and computer program product of the present invention, the coarse face region in an image can be precisely segmented, which provides a good basis for the subsequent image processing based on the fine face segmentation region.

Claims

1. A method, comprising:

performing face detection of an image;

obtaining a coarse face segmentation region of at least one face and a contour edge of the at least one face based on the face detection;

searching the contour edge within a preset position adjusting range in a lateral or longitudinal direction to locate a plurality of updated contour edge points, wherein the preset position adjusting range is based upon a two dimensional coordinate of (x,y) and includes a lateral search range of (x−

δ

, x+δ

) and a longitudinal search range of (y−

δ

, y+δ

), wherein δ

varies from one two dimensional coordinate to another, and wherein δ

is selected so as to define a weight of contour edge information; and

adjusting the coarse face segmentation region based on the contour edge, the weight of the contour edge information as defined by δ

, and the plurality of updated contour edge points to obtain a fine face segmentation region.

2. The method of claim 1, wherein said obtaining a coarse face segmentation region of at least one face comprises:

building a skin color model by using a partial region of the at least one face; and

applying the skin color model to the at least one face to obtain the coarse face segmentation region of the at least one face.

3. The method of claim 1, wherein the contour edge of the at least one face is obtained by using a wavelet transform.

4. The method of claim 1, wherein said adjusting the coarse face segmentation region based on the contour edge to obtain a fine face segmentation region comprises:

adjusting the coarse face segmentation region along at least one of the lateral direction and the longitudinal direction with the contour edge as a reference, so as to fill the whole contour with the coarse face segmentation region.

5. The method of claim 4, wherein said adjusting the coarse face segmentation region comprises:

detecting whether edge pixels of the coarse face segmentation region deviate from the contour edge within the preset position adjusting range; and

when detecting that the deviation is within the position adjusting range, adjusting the deviated edge pixels to points on the contour edge aligning with these pixels in the lateral or longitudinal direction.

6. The method of claim 5, further comprising:

detecting whether the edge pixels are protruding outward from the contour edge and exceeding the position adjusting range; and

when detecting that the protrusion exceeds the position adjusting range, removing a portion of the coarse face segmentation region protruding from the contour edge.

7. The method of claim 6, further comprising:

detecting whether the contour edge is disconnected; and

when detecting that the contour edge is disconnected, performing an interpolation operation between the disconnected two end points.

8. The method of claim 1, further comprising:

determining grey values of all pixels within the fine face segmentation region;

dividing all pixels into at least two classes based on the grey values; and

adjusting the grey values of each class of pixels according to different levels, so as to realize whitening processing on the fine face segmentation region.

9. The method of claim 8, wherein said dividing all pixels into at least two classes based on the grey values comprises:

taking statistics of the grey values of all pixels within the fine face segmentation region;

determining at least one threshold value for the grey values based on the statistics; and

dividing all pixels into at least two classes based on the at least one threshold value.

10. The method of claim 9, wherein said determining at least one threshold value for the grey values comprises:

ranking the grey values of all pixels;

selecting a predetermined number of pixels in sequence; and

averaging the grey values of the selected pixels to determine the at least one threshold value.

11. The method of claim 8, further comprising:

selecting a plurality of different levels for each class of pixels for the adjustment, so as to realize the whitening processing with a plurality of corresponding different effects.

12. The method of claim 1, further comprising:

determining grey value differences between each pixel within the fine face segmentation region and respective neighboring pixels in a neighboring region;

comparing each of the grey value differences with the predetermined threshold value to determine smoothing weights of the each pixel over respective neighboring pixels, wherein the smoothing weight is inversely proportional to the grey value difference; and

adjusting the grey value of the each pixel based on the grey values and the smoothing weights of the respective neighboring pixels, and spatial distances between the respective neighboring pixels and the each pixel, so as to realize smoothing processing on the fine face segmentation region.

13. The method of claim 12, wherein the predetermined threshold value is obtained by the following operations:

calculating gradient values of all pixels within the fine face segmentation region;

ranking the gradient values of all pixels;

selecting a predetermined number of pixels in sequence; and

averaging the gradient values of the selected pixels to obtain the predetermined threshold value.

14. An apparatus, comprising:

at least one processor and at least one memory containing computer program code;

the processor and the memory are configured to, with the processor, cause the apparatus to at least;

perform face detection of an image;

obtain a coarse face segmentation region of at least one face and a contour edge of the at least one face based on the face detection;

search the contour edge within a preset position adjusting range in a lateral or longitudinal direction to locate a plurality of updated contour edge points, wherein the preset position adjusting range is based upon a two dimensional coordinate of (x, y) and includes a lateral search range of (x−

δ

, x+δ

) and a longitudinal search range of (y−

δ

, y+δ

), wherein δ

varies from one two dimensional coordinate to another, and wherein δ

is selected so as to define a weight of contour edge information; and

adjust the coarse face segmentation region based on the contour edge, the weight of the contour edge information as defined by δ

, and the plurality of updated contour edge points to obtain a fine face segmentation region.

15. The apparatus of claim 14, wherein the processor and the memory are configured to, with the processor, cause the apparatus to at least:

build a skin color model by using a partial region of the at least one face; and

apply the skin color model to the at least one face to obtain the coarse face segmentation region of the at least one face.

16. The apparatus of claim 14, wherein the contour edge of the at least one face is obtained by using a wavelet transform.

17. The apparatus of claim 14, wherein the processor and the memory are configured to, with the processor, cause the apparatus to at least:

adjust the coarse face segmentation region along at least one of the lateral direction and the longitudinal direction with the contour edge as a reference, so as to fill the whole contour with the coarse face segmentation region.

18. The apparatus of claim 17, wherein the processor and the memory are configured to, with the processor, cause the apparatus to at least:

detect whether edge pixels of the coarse face segmentation region deviate from the contour edge within the preset position adjusting range; and

when detecting that the deviation is within the position adjusting range, adjust the deviated edge pixels to points on the contour edge aligning with these pixels in the lateral or longitudinal direction.

19. The apparatus of claim 18, wherein the processor and the memory are configured to, with the processor, cause the apparatus to further:

detect whether the edge pixels are protruding outward from the contour edge and exceeding the position adjusting range; and

when detecting that the protrusion exceeds the position adjusting range, remove a portion of the coarse face segmentation region protruding from the contour edge.

20. The apparatus of claim 19, wherein the processor and the memory are configured to, with the processor, cause the apparatus to further:

detect whether the contour edge is disconnected; and

when detecting that the contour edge is disconnected, perform an interpolation operation between the disconnected two end points.

21. The apparatus of claim 14, wherein the processor and the memory are configured to, with the processor, cause the apparatus to further:

determine grey values of all pixels within the fine face segmentation region;

divide all pixels into at least two classes based on the grey values; and

adjust the grey values of each class of pixels according to different levels, so as to realize whitening processing on the fine face segmentation region.

22. The apparatus of claim 21, wherein the processor and the memory are configured to, with the processor, cause the apparatus to at least:

take statistics of the grey values of all pixels within the fine face segmentation region;

determine at least one threshold value for the grey values based on the statistics;

divide all pixels into at least two classes based on the at least one threshold value.

23. The apparatus of claim 22, wherein the processor and the memory are configured to, with the processor, cause the apparatus to at least:

rank the grey values of all pixels;

select a predetermined number of pixels in sequence; and

average the grey values of the selected pixels to determine the at least one threshold value.

24. The apparatus of claim 21, wherein the processor and the memory are configured to, with the processor, cause the apparatus to at least:

select a plurality of different levels for each class of pixels for the adjustment, so as to realize the whitening processing with a plurality of corresponding different effects.

25. The apparatus of claim 14, wherein the processor and the memory are configured to, with the processor, cause the apparatus to further:

determine grey value differences between each pixel within the fine face segmentation region and respective neighboring pixels in a neighboring region;

compare each of the grey value differences with the predetermined threshold value to determine smoothing weights of the each pixel over respective neighboring pixels, wherein the smoothing weight is inversely proportional to the grey value difference; and

adjust the grey value of the each pixel based on the grey values and the smoothing weights of the respective neighboring pixels, and spatial distances between the respective neighboring pixels and the each pixel, so as to realize smoothing processing on the fine face segmentation region.

26. The apparatus of claim 25, wherein the processor and the memory are configured to, with the processor, cause the apparatus to obtain the predetermined threshold value by performing at least the following operations:

computing gradient values of all pixels within the fine face segmentation region;

ranking the gradient values of all pixels;

selecting a predetermined number of pixels in sequence; and

averaging the gradient values of the selected pixels to obtain the predetermined threshold value.

27. A non-transitory computer program product comprising at least one computer-readable storage medium having computer-executable program code portions stored therein, the computer-executable program code portions comprising:

program code instructions causing performing face detection of an image;

program code instructions causing obtaining a coarse face segmentation region of at least one face and a contour edge of the at least one face based on the face detection;

program code instructions causing searching the contour edge within a preset position adjusting range in a lateral or longitudinal direction to locate a plurality of updated contour edge points, wherein the preset position adjusting range is based upon a two dimensional coordinate of (x, y) coarse face segmentation region and includes a lateral search range of (x−

δ

, x+δ

) and a longitudinal search range of (y−

δ

, y+δ

), wherein δ

varies from one two dimensional coordinate to another, and wherein δ

is selected so as to define a weight of contour edge information; and

program code instructions causing adjusting the coarse face segmentation region based on the contour edge, the weight of the contour edge information as defined by δ

, and the plurality of updated contour edge points to obtain a fine face segmentation region.