Methods and Apparatus for Facial Feature Replacement

• US 20130129141A1
• Filed: 11/24/2010
• Published: 05/23/2013
• Est. Priority Date: 08/20/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wang, Jue, Shechtman, Elya, Bourdev, Lubomir D., Yang, Fei
• Granted Patent: US 8,818,131 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 2210/44 Morphing

Abstract

Three dimensional models corresponding to a target image and a reference image are selected based on a set of feature points defining facial features in the target image and the reference image. The set of feature points defining the facial features in the target image and the reference image are associated with corresponding 3-dimensional models. A 3D motion flow between the 3-dimensional models is computed. The 3D motion flow is projected onto a 2D image plane to create a 2D optical field flow. The target image and the reference image are warped using the 2D optical field flow. A selected feature from the reference image is copied to the target image.

Claims

1. A computer-implemented method, comprising:

selecting three-dimensional models corresponding to a target image and a reference image based on a set of feature points defining features in the target image and the reference image;

associating the set of feature points defining the features in the target image and the reference image with the three-dimensional models;

computing a 3D motion flow between the three-dimensional models;

projecting the 3D motion flow onto a 2D image plane to create a 2D optical field flow;

warping the target image and the reference image using the 2D optical field flow; and

copying a selected feature from the reference image to the target image.

2. The computer-implemented method of claim 1, further comprising blending the selected feature into the target image.

3. The computer-implemented method of claim 1, wherein the selecting comprises scoring each model of a set of potential models for its correspondence to the reference image and the target image based on locations of feature points on the each model of the set of potential models and locations of corresponding feature points from the set of feature points defining the features in the target image and the reference image.

4. The computer-implemented method of claim 1, further comprising correcting illumination to achieve shadows and to normalize differences in position, intensity and color temperature of light applied to photographic subjects during creation of the reference image and the target image.

5. The computer-implemented method of claim 1, further comprising detecting in the target image and the reference image the set of feature points for defining the features in the target image and the reference image.

6. The computer-implemented method of claim 1, wherein the warping the target image comprises warping the target image to accommodate a shape of a face in the target image to receive the feature being imported from the reference image.

7. The computer-implemented method of claim 1, wherein the warping the reference image further comprises warping the reference image to correct the orientation of the selected feature to match an orientation of the target image.

8. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

select three dimensional models corresponding to a target image and a reference image based on a set of feature points defining features in the target image and the reference image,

associate the set of feature points defining the features in the target image and the reference image with the three-dimensional models,

compute a 3D motion flow between the three-dimensional models,

project the 3D motion flow onto a 2D image plane to create a 2D optical field flow,

warp the target image and the reference image using the 2D optical field flow, and

copy a selected feature from the reference image to the target image.

9. The system of claim 8, wherein the program instructions are further executable by the at least one processor to blend the selected feature into the target image.

10. The system of claim 8, wherein the program instructions executable by the at least one processor to select the three dimensional models further comprise program instructions executable by the at least one processor to score each model of a set of potential models for its correspondence to the reference image and the target image based on locations of feature points on the each model of the set of potential models and locations of corresponding feature points from the set of feature points defining the features in the target image and the reference image.

11. The system of claim 8, wherein the program instructions are further executable by the at least one processor to correct illumination to achieve shadows and to normalize differences in position, intensity and color temperature of light applied to photographic subjects during creation of the reference image and the target image.

12. The system of claim 8, wherein the program instructions are further executable by the at least one processor to detect in the target image and the reference image the set of feature points for defining the features in the target image and the reference image.

13. The system of claim 8, wherein the program instructions executable by the at least one processor to warp the target image further comprise program instructions executable by the at least one processor warp the target image to accommodate a shape of a face in the target image to receive the feature being imported from the reference image.

14. The system of claim 8, wherein the program instructions executable by the at least one processor to warp the reference image further comprise program instructions executable by the at least one processor warp the reference image to correct the orientation of the selected feature to match an orientation of the target image.

15. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

selecting three dimensional models corresponding to a target image and a reference image based on a set of feature points defining features in the target image and the reference image;

associating the set of feature points defining the features in the target image and the reference image with the three-dimensional models;

computing a 3D motion flow between the three-dimensional models;

projecting the 3D motion flow onto a 2D image plane to create a 2D optical field flow;

warping the target image and the reference image using the 2D optical field flow; and

copying a selected feature from the reference image to the target image.

16. The non-transitory computer-readable storage medium of claim 15, further comprising program instructions that are computer-executable to implement blending the selected feature into the target image.

17. The non-transitory computer-readable storage medium of claim 15, wherein program instructions that are computer-executable to implement the selecting the three-dimensional models further comprise program instructions that are computer-executable to implement scoring each model of a set of potential models for its correspondence to the reference image and the target image based on locations of feature points on the each model of the set of potential models and locations of corresponding feature points from the set of feature points defining the features in the target image and the reference image.

18. The non-transitory computer-readable storage medium of claim 15, further comprising program instructions that are computer-executable to implement correcting illumination to achieve shadows and to normalize differences in position, intensity and color temperature of light applied to photographic subjects during creation of the reference image and the target image.

19. The non-transitory computer-readable storage medium of claim 15, further comprising program instructions that are computer-executable to implement detecting in the target image and the reference image the set of feature points for defining the features in the target image and the reference image.

20. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions that are computer-executable to implement the warping the target image further comprise program instructions that are computer-executable to implement warping the target image to accommodate a shape of a face in the target image to receive the feature being imported from the reference image.