Methods and apparatus for facial feature replacement

• US 8,818,131 B2
• Filed: 11/24/2010
• Issued: 08/26/2014
• Est. Priority Date: 08/20/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wang, Jue, Shechtman, Elya, Bourdev, Lubomir D., Yang, Fei
• Primary Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US12/954,487
• Publication Number:
  US 20130129141A1
• Time in Patent Office: 1,371 Days
• Field of Search: None
• US Class Current: 382/285
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/44 Morphing

Abstract

Three dimensional models corresponding to a target image and a reference image are selected based on a set of feature points defining facial features in the target image and the reference image. The set of feature points defining the facial features in the target image and the reference image are associated with corresponding 3-dimensional models. A 3D motion flow between the 3-dimensional models is computed. The 3D motion flow is projected onto a 2D image plane to create a 2D optical field flow. The target image and the reference image are warped using the 2D optical field flow. A selected feature from the reference image is copied to the target image.

Claims

1. A computer-implemented method, comprising:

receiving a selection of a feature in a reference image to include in a target image;

selecting, from a set of potential models, a first three-dimensional model corresponding to the target image based on a set of feature points detected in the target image;

selecting, from the set of potential models, a second three-dimensional model corresponding to the reference image based on a set of feature points detected in the reference image;

associating the set of feature points detected in the target image and the reference image with the first and second three-dimensional models, respectively;

computing a 3D motion flow between the first and second three-dimensional models;

responsive to the computing, projecting the 3D motion flow onto a 2D image plane to create a 2D optical field flow;

responsive to the projecting, warping the target image based, at least in part, on the 2D optical field flow;

responsive to the projecting, warping the reference image based, at least in part, on the 2D optical field flow; and

responsive to warping the target image and warping the reference image, copying the selected feature from the reference image to the target image.

2. The computer-implemented method of claim 1, further comprising blending the selected feature into the target image.

3. The computer-implemented method of claim 1, wherein the selecting of the first and second three-dimensional models comprises scoring each model of the set of potential models for its correspondence to the reference image and the target image based on locations of feature points on the each model of the set of potential models and locations of corresponding feature points from the set of feature points detected in the target image and the reference image.

4. The computer-implemented method of claim 1, further comprising correcting illumination to achieve shadows and to normalize differences in position, intensity and color temperature of light applied to photographic subjects of the reference image and the target image.

5. The computer-implemented method of claim 1, wherein the warping the target image comprises accommodating a shape of a face in the target image to receive the feature selected from the reference image.

6. The computer-implemented method of claim 1, wherein the warping the reference image further comprises warping the reference image to correct the orientation of the selected feature to match an orientation of the target image.

7. A system, comprising:

at least one processor; and

a memory comprising program instructions, the program instructions being executable by the at least one processor to;

receive selection of a feature in a reference image to include in a target image;

responsive to the selection of the feature in the reference image, select a first three-dimensional model corresponding to the target image based on a set of feature points defining features in the target image;

responsive to the selection of the feature in the reference image, select a second three-dimensional model corresponding to the reference image based on a set of feature points detected in the reference image;

compute a 3D motion flow between the first and second three-dimensional models;

warp the target image and the reference image based, at least in part, on the computed 3D motion flow; and

copy the selected feature from the warped reference image to the warped target image.

8. The system of claim 7, wherein the program instructions are further executable by the at least one processor to blend the selected feature into the target image.

9. The system of claim 7, wherein the program instructions are further executable by the at least one processor to score each model of a set of potential models for correspondence to the reference image and the target image based on comparing locations of feature points on each respective model and locations of corresponding feature points from the set of feature points defining the features in the target image and the set of feature points in the reference image.

10. The system of claim 7, wherein the program instructions are further executable by the at least one processor to correct illumination to achieve shadows and to normalize differences in position, intensity and color temperature of light applied to photographic subjects of the reference image and the target image.

11. The system of claim 7, wherein the program instructions are further executable by the at least one processor to detect in the target image and the reference image the set of feature points for defining the features in the target image and the reference image.

12. The system of claim 7, wherein the program instructions are further executable by the at least one processor to warp the target image to accommodate a shape of a face in the target image to receive the feature selected from the reference image.

13. The system of claim 7, wherein the program instructions are further executable by the at least one processor to warp the reference image to correct the orientation of the selected feature to match an orientation of the target image.

14. A computer-readable storage device storing program instructions, the program instructions being computer-executable to perform operations comprising:

receiving a selection of a feature in a reference image to include in a target image;

responsive to the receiving, selecting a first three dimensional model corresponding to the target image based on a set of feature points defining features in the target image;

responsive to the receiving, selecting a second three-dimensional model corresponding to the reference image based on the set of feature points defining features in the reference image;

responsive to selecting the first and second three-dimensional models, computing a 3D motion flow between the first and second three-dimensional models;

responsive to the computing, warping the reference image based, at least in part, on the computed 3D motion flow;

responsive to the computing, warping the target image to accommodate the selected feature from the reference image based, at least in part, on the computed 3D motion flow; and

copying the selected feature from the warped reference image to the warped target image.

15. The computer-readable storage device of claim 14, the operations further comprising blending the selected feature into the target image.

16. The non-transitory computer-readable storage medium of claim 14, wherein the selecting the first and second three-dimensional models comprises scoring each model of a set of potential models for its correspondence to the reference image and the target image based on locations of feature points on the each model of the set of potential models and locations of corresponding feature points from the set of feature points defining the features in the target image and the reference image.

17. The computer-readable storage device of claim 14, the operations further comprising correcting illumination to achieve shadows and to normalize differences in position, intensity and color temperature of light applied to photographic subjects of the reference image and the target image.

18. The computer-readable storage device of claim 14, the operations further comprising detecting in the target image and the reference image the set of feature points for defining the features in the target image and the reference image.

19. The computer-readable storage device of claim 14, the operations further comprising warping the target image to accommodate a shape of a face in the target image to receive the feature selected from the reference image.

20. The computer-readable storage device of claim 14, the operations further comprising:

projecting the three-dimensional motion flow onto a two-dimensional plane; and

warping the target image based on the projection of the three-dimensional motion flow onto the two-dimensional plane.