Methods and Apparatus for Automated Portrait Retouching Using Facial Feature Localization

• US 20130044947A1
• Filed: 07/31/2012
• Published: 02/21/2013
• Est. Priority Date: 08/19/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Brandt, Jonathan W., Lin, Zhe, Le, Vuong
• Granted Patent: US 8,811,686 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

Various embodiments of methods and apparatus for facial retouching are disclosed. In one embodiment, a face in an input image is detected. Independent sets of feature points are detected for respective facial feature components. A plurality of masks for each of the facial feature components is generated. Using the plurality of masks, retouch effects are performed to the facial feature components. Some embodiments provide for user interaction to constrain the mask generation.

Claims

1. A method, comprising:

performing, using one or more processors,

detecting a face in an input image;

generating a plurality of masks for each of a plurality of facial feature components by detecting independent sets of feature points for respective facial feature components; and

using the plurality of masks, performing retouch effects to the facial feature components.

2. The method of claim 1, wherein the detecting the independent sets of feature points for the respective facial feature components further comprises detecting the independent sets of feature points for the respective facial feature components using a component-based active shape model for facial feature localization.

3. The method of claim 1, wherein the generating the plurality of masks for each of the facial feature components further comprises:

presenting the feature point locations through a user interface;

receiving user modifications of the feature point locations; and

modifying ones of the plurality of masks based on the user modifications.

4. The method of claim 1, wherein the generating the plurality of masks further comprises:

rendering a closed region into a selected mask as a set of ones;

eroding the closed region by an amount proportional to a size of the closed region;

dilating the closed region;

executing a graph cut operation to generate a buffer for the closed region; and

blurring the buffer.

5. The method of claim 1, wherein the generating the plurality of masks further comprises generating a skin mask with a skin probability for each of a set of pixels.

6. The method of claim 1, wherein the performing retouch effects to the facial feature components further comprises adjusting one or more facial component shapes based on the plurality of masks.

7. The method of claim 1, wherein the performing retouch effects to the facial feature components further comprises adjusting one or more facial component colors based on the plurality of masks.

8. The method of claim 1, wherein the detecting the independent sets of feature points for the respective facial feature components further comprises detecting the independent sets of feature points for the respective facial feature components using user input as a constraint.

9. The method of claim 1, wherein the detecting the independent sets of feature points for the respective facial feature components further comprises detecting the independent sets of feature points for the respective facial feature components using a component-based approach.

10. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

detecting a face in an input image;

detecting independent sets of feature points for respective facial feature components;

generating a plurality of masks for each of the facial feature components; and

using the plurality of masks, performing retouch effects to the facial feature components.

11. The system of claim 10, wherein the program instructions executable by the at least one processor to detect the independent sets of feature points for the respective facial feature components further comprise program instructions executable by the at least one processor to detect the independent sets of feature points for the respective facial feature components using a component-based active shape model for facial feature localization.

12. The system of claim 10, wherein the program instructions executable by the at least one processor to generate the plurality of masks for each of the facial feature components further comprise program instructions executable by the at least one processor to:

present the feature point locations through a user interface;

receive user modifications of the feature point locations; and

modify ones of the plurality of masks based on the user modifications.

13. The system of claim 10, wherein the program instructions executable by the at least one processor to generate the plurality of masks further comprise program instructions executable by the at least one processor to:

render a closed region into a selected mask as a set of ones;

erode the closed region by an amount proportional to a size of the closed region;

dilate the closed region;

execute a graph cut operation to generate a buffer for the closed region; and

blur the buffer.

14. The system of claim 10, wherein the program instructions executable by the at least one processor to generate the plurality of masks further comprise program instructions executable by the at least one processor to generate a skin mask with a skin probability for each of a set of pixels.

15. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

detecting a face in an input image;

detecting independent sets of feature points for respective facial feature components;

generating a plurality of masks for each of the facial feature components; and

using the plurality of masks, performing retouch effects to the facial feature components.

16. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to implement detecting the independent sets of feature points for the respective facial feature components further comprise program instructions computer-executable to implement detecting the independent sets of feature points for the respective facial feature components using a component-based active shape model for facial feature localization.

17. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to implement the generating the plurality of masks for each of the facial feature components further comprise program instructions computer-executable to implement:

presenting the feature point locations through a user interface;

receiving user modifications of the feature point locations; and

modifying ones of the plurality of masks based on the user modifications.

18. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to implement the generating the plurality of masks further comprise program instructions computer-executable to implement:

rendering a closed region into a selected mask as a set of ones;

eroding the closed region by an amount proportional to a size of the closed region;

dilating the closed region;

executing a graph cut operation to generate a buffer for the closed region; and

blurring the buffer.

19. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to implement the generating the plurality of masks further comprise program instructions computer-executable to implement generating a skin mask with a skin probability for each of a set of pixels.

20. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to implement the performing retouch effects to the facial feature components further comprise program instructions computer-executable to implement adjusting one or more facial component shapes based on the plurality of masks.