Recomposing photographs from multiple frames

• US 20070237421A1
• Filed: 03/29/2006
• Published: 10/11/2007
• Est. Priority Date: 03/29/2006
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Luo, Jiebo, Lei, Phoury
• Granted Patent: US 7,787,664 B2
• Time in Patent Office: Days
• US Class Current: 382/284
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A method for replacing a face in a first digital image with a face from a second digital image including automatically detecting one or more faces in the first digital image; identifying at least one target face from the detected faces that needs to be replaced; automatically detecting one or more faces in the second digital image; identifying at least one source face from the second image detected faces for replacing the target face; using features from the target and source faces to perform facial geometry and appearance corrections to the source face; and replacing the target face with the corrected source face and blending the corrected source face into the first digital image.

Claims

1. A method for replacing a face in a first digital image with a face from a second digital image comprising:

a. automatically detecting one or more faces in the first digital image;

b. identifying at least one target face from the detected faces that needs to be replaced;

c. automatically detecting one or more faces in the second digital image;

d. identifying at least one source face from the second image detected faces for replacing the target face;

e. using features from the target and source faces to perform facial geometry and appearance corrections to the source face; and

f. replacing the target face with the corrected source face and blending the corrected source face into the first digital image.

2. The method according to claim 1 wherein step d includes using facial features in the target face to identify the source face through automatic face recognition.

3. The method according to claim 1 further including

g. interactively fine tuning the replaced source face using parameters automatically determined in steps e and f as starting points.

4. The method of claim 1 further including

i. performing face geometric correction on the source face using facial features located on the target face and the source face;

ii. performing appearance correction on the geometrically corrected source face to produce a corrected source face;

iii. producing a source face mask for the corrected source face;

iv. applying morphological erosion to the source face mask to produce an eroded source face mask;

v. replacing pixels in the target face with pixels in the corrected source face based on the eroded source face mask; and

vi. blending the corrected source face into the first image around the boundary pixels indicated by the eroded source face mask.

5. The method according to claim 3 further including

i. providing a plurality of sliders on a pop-up panel, wherein the sliders correspond to parameters related to brightness, contrast, color along the magenta-green axis (c1=r−

2*g+b), color along the red and blue axis (c2=r−

b), face size, horizontal translation, vertical translation, and a smudge kernel for smoothing the boundaries; and

ii. using user input from a slider to fine tune the quality of the composite image through changing the corresponding parameter, using automatically determined parameter as a starting point.

6. The method according to claim 3 further including

i. providing an interface to allow a user to click on two eyes of a subject in case automatic face detection fails to find the face;

and

ii. using the user clicked eye positions to locate face contour and other facial features.

7. A method for producing a composite group image from a series of images of the group captured by an image capture device, comprising:

a. automatically detecting faces in the group from a first digital image in the series;

b. automatically detecting the group faces in a second digital image;

c. automatically identifying at least one target face from the detected group faces that needs to be replaced;

d. using facial features in the target face to identify a source face in the second digital image through automatic face recognition;

e. using features from the target and source faces to perform facial geometry and appearance corrections to the source face; and

f. replacing the target face with the corrected source face and blending the corrected source face into the first digital image to form the composite group image.

8. The method of claim 7 wherein there are more than two digital images and wherein steps c-f are repeated for different persons in the group.

9. The method of claim 7 further including

g. interactively fine tuning the replaced source face using parameters automatically determined in steps e and f as starting points.

10. A method for producing a composite image from a series of frames captured by a video capture device, comprising:

a. selecting a frame from the series of frames;

b. automatically detecting one or more faces from the selected frame;

c. automatically identifying at least one target face from the detected group faces that needs to be replaced;

d. automatically detecting faces in other frames captured by the video capture device;

e. using facial features in the target face to identify source faces in the other frames through automatic face recognition;

f. producing a collage of usable source faces from the identified source faces in the other frames;

g. selecting a source face from the collage of useable source faces;

h. using features from the target and the selected source face to perform facial geometry and appearance corrections to the selected source face; and

i. replacing the target face with the corrected source face and blending the corrected source face into the first digital image to form the composite group image.

11. The method of claim 10 further including

j. interactively fine tuning the replaced source face using parameters automatically determined in steps e and f as starting points.