IMAGE MASKS FOR FACE-RELATED SELECTION AND PROCESSING IN IMAGES

• US 20140341442A1
• Filed: 05/14/2013
• Published: 11/20/2014
• Est. Priority Date: 05/14/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lewis, Justin, Krishnaswamy, Aravind, James, Gavin, Lang, Grayson, Babacan, Sevket Derin
• Granted Patent: US 8,983,152 B2
• Time in Patent Office: Days
• US Class Current: 382/118
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

In some implementations, a method includes identifying one or more face regions of an image, the face regions including pixels that depict at least a portion of one or more faces of persons. The face regions are identified based on identifying facial landmarks of the faces. The method determines an associated face mask for each of the faces based on the face regions, where each face mask indicates which pixels in the image depict the corresponding face. Face pixels can be selected for processing by applying the face masks, and image pixels outside the faces can be selected by inversely applying the face masks. The selected pixels can be provided to a processing operation for adjustment of the selected pixels.

Claims

1. A method comprising:

identifying one or more face regions of an image, the one or more face regions including pixels that depict at least a portion of one or more corresponding faces of one or more persons, wherein identifying the one or more face regions is based on identifications of facial landmarks of the one or more faces;

determining an associated face mask for each of the one or more faces based on the one or more face regions, wherein each face mask indicates which pixels in the image depict the corresponding face;

selecting the pixels of the image outside the one or more faces by inversely applying the one or more face masks; and

providing the selected pixels to a processing operation for adjustment of the selected pixels.

2. The method of claim 1 wherein determining an associated face mask includes adding connected face pixels to each face region, wherein the associated face mask includes approximately the entire corresponding face of a person.

3. The method of claim 2 wherein each of the one or more face regions includes eyes and mouth of one of the persons, and wherein a blobbing technique adds contiguous skin pixels outside each face region to each face region, the skin pixels representing skin of one of the persons, wherein the blobbing technique examines a color of pixels outside each of the face regions and adds pixels to each face region in response to the color being within a predetermined threshold of a color of one or more other pixels adjacent to the pixels.

4. The method of claim 1 wherein determining an associated face mask includes:

determining an average color of a plurality of pixels within the face region,

determining a spatial face area approximately defining the corresponding face of the associated person; and

selecting pixels in the spatial face area that are similar in color to the average color and designating in the face mask that the selected pixels are face pixels.

5. The method of claim 4 wherein the plurality of pixels for which the average color is determined are pixels selected as being within a predetermined range of known skin colors.

6. The method of claim 4 wherein determining a spatial face mask area includes estimating width and height for the spatial face mask area based on predetermined average facial proportions of a plurality of people.

7. The method of claim 1 further comprising providing a falloff region for the face mask in which pixels are designated to be processed by a gradually lessening amount in a direction from a face area to the portions of the image outside the face area.

8. The method of claim 1 wherein the processing operation includes at least one of:

applying a Gaussian blur to the selected pixels; and

applying a local contrast enhancement to the selected pixels.

9. A method comprising:

identifying a face region within an image, the face region including pixels that depict at least a portion of a face of a person;

selecting a plurality of pixels within the face region that are within a predetermined range of colors known to be skin colors;

determining a characteristic skin color for the selected pixels in the face region; and

determining a face mask for the image, the face mask indicating which pixels in the image are skin pixels depicting the person'"'"'s skin based on a color threshold similarity to the characteristic skin color.

10. The method of claim 9 wherein the characteristic skin color is an average color of the selected pixels in the face region.

11. The method of claim 9 further comprising determining an indication of a distribution of colors of the selected pixels in the face region, wherein the distribution is used in determining a color distance threshold used in the color threshold similarity to the characteristic skin color.

12. The method of claim 11 wherein the indication of a distribution of the colors of the pixels includes a standard deviation.

13. The method of claim 9 wherein the face region is identified based on facial recognition of one or more facial landmarks of the face.

14. The method of claim 9 further comprising determining a spatial face area in the image based on facial landmarks provided from the identified face region, the spatial face area approximately defining the face of the person, wherein the skin pixels in the face mask are limited to pixels within the spatial face area.

15. The method of claim 14 wherein determining a face mask includes determining a falloff area outside the spatial face area, wherein the pixels in the falloff area are evaluated for similarity to the characteristic skin color and are designated to have a gradation of less processing than the pixels in the spatial face area.

16. The method of claim 9 wherein each pixel in the face mask indicates a degree to which a corresponding pixel in the image is similar to the characteristic skin color and a degree of processing for the corresponding pixel.

17. The method of claim 9 wherein selecting a plurality of pixels includes converting colors of the pixels in the face region to a different color space, and checking which pixels in the face region have colors included in at least one predetermined range of the color space indicative of the known skin colors, wherein pixels having colors outside the predetermined area are excluded from the selection.

18. The method of claim 9 further comprising adjusting the skin pixels of the image as designated by the face mask, wherein the adjustment includes a blurring filter to smooth one or more depicted features of the face in the image.

19. The method of claim 9 further comprising inverting the face mask and adjusting the pixels that are not skin pixels of the face mask to process the image.

20. A system comprising:

a storage device; and

at least one processor accessing the storage device and operative to perform operations comprising;

identifying a face region within an image, the face region including pixels that depict at least a portion of a face of a person;

selecting a plurality of pixels within the face region that are within a predetermined range of colors known to be skin colors;

determining a characteristic skin color for the selected pixels in the face region; and

determining a face mask for the image, the face mask indicating which pixels in the image are skin pixels depicting the person'"'"'s skin based on a color threshold similarity to the characteristic skin color.