Method and system for generating a foreground mask for a composite image

• US 7,024,054 B2
• Filed: 09/27/2002
• Issued: 04/04/2006
• Est. Priority Date: 09/27/2002
• Status: Active Grant

• Current Assignee: Monument Peak Ventures, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Eastman Kodak Company
• Inventors: Luo, Jiebo, Parulski, Kenneth A., Cahill, Nathan D., Gindele, Edward B.
• Primary Examiner(s): Shah, Sanjiv
• Assistant Examiner(s): Bayat, Ali
• Application Number:
  US10/256,900
• Publication Number:
  US 20040062439A1
• Time in Patent Office: 1,285 Days
• Field of Search: 382/103, 382/218, 382/173, 382/162, 382/145, 382/209, 382/260, 382/262, 382/154, 382/132, 382/180, 382/294, 348/239, 348/564, 348/231.99, 348/854, 348/586, 348/587, 345/419, 345/589, 345/634, 345/590, 345/592, 375/240.12, 359/743
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A technique for producing a foreground mask from a digital image is based on receiving two digital images, wherein a first digital image is derived from a first original scene image including a foreground photographed against a background having arbitrary and unknown content matter, and a second digital image is derived from a second original scene image including substantially the same background but without the foreground. The second digital image is automatically aligned to the first digital image without user intervention to generate an aligned digital image including the background without the foreground. Then, using the first digital image and the aligned digital image, a foreground mask is produced relating to the spatial regions in the first digital image pertaining to the foreground, whereby the foreground mask is subsequently used to generate a composite digital image.

Claims

1. A method for producing a foreground mask from a digital image, said method comprising the steps of:

a) receiving two digital images, each having a plurality of pixels, wherein a first digital image is derived from a first original scene image including a foreground photographed against a background having arbitrary and unknown content matter, and a second digital image is derived from a second original scene image including substantially the same background but without the foreground;

b) automatically aligning the second digital image to the first digital image to generate an aligned digital image including the background without the foreground, wherein the automatic aligning is achieved without user intervention; and

c) using the first digital image and the aligned digital image to produce a foreground mask relating to an area in the first digital image pertaining to the foreground, whereby the foreground mask is subsequently used to generate a composite digital image.

2. The method of claim 1, wherein the step (c) of using the first digital image and aligned digital image to produce a foreground mask comprises the steps of:

i) subtracting the aligned digital image from the first digital image to generate a difference image;

ii) processing the difference image to generate a mask image;

iii) applying the mask image to the first digital image to select an area in which the foreground is located, whereby the selected foreground area is used in the compositing of a new image with a different background.

3. The method of claim 2, wherein the first and second digital images have been captured with a digital camera that includes a display device and further including the steps of:

d) displaying the composite digital image on the display device;

e) selecting a different position for the foreground area within composite digital image; and

;

f) producing a new composite image responsive to the selected different position.

4. The method of claim 2, wherein the first and second digital images have been captured with a digital camera that includes a display device and further including the steps of:

d) displaying the composite digital image on the display device;

e) selecting a different size for the foreground area relative to the size of the composite digital image; and

;

f) producing a new composite image responsive to the selected different size.

5. The method of claim 1, wherein the step (a) of receiving two digital images comprises the steps of:

i) capturing and displaying the first digital image;

ii) displaying and, with user intervention, coarsely aligning a live image with the first digital image; and

iii) capturing the live image to yield the second digital image;

wherein the second digital image is coarsely adjusted by the user to include substantially the same background matter without the subject.

6. The method of claim 1, wherein the step (a) of receiving the two digital images comprises the steps of:

i) initiating a composite image capture process in a digital camera by capturing and storing the first digital image and displaying a portion of the first digital image that is intended to overlap with the second digital image; and

ii) displaying a live image in an overlapping relationship with the first digital image to enable a user to coarsely align the live image with the first digital image before capturing the live image as the second digital image.

7. A method for producing a subject mask from a digital image for use in a compositing process, said method comprising the steps of:

a) receiving two digital images, each having a plurality of pixels, wherein a first digital image is derived from a first original scene image including a subject photographed against a background having arbitrary and unknown content matter, and a second digital image is derived from a second original scene image including substantially the same background matter without the subject; and

,

b) automatically processing the second digital image in relation to the first digital image to generate an aligned digital image including the background matter without the subject, wherein the automatic processing is achieved through a homographic transformation without user intervention; and

,

c) using the first digital image and the aligned digital image to produce a subject mask relating to an area in the first digital image pertaining to the subject, whereby the subject mask is used to generate a composite digital image.

8. The method as claimed in claim 7 wherein the step b) of automatically aligning the second digital image to the first digital image comprises the steps of:

i) choosing a set of points in the first digital image and locating their corresponding points in the second digital image using a similarity measure, thereby generating a collection of points;

ii) for each prospective point in the collection, selecting a matching point in the second digital image based on a neighborhood similarity measure evaluated within an area centered on the prospective point in the collection, thereby generating a collection of matching points representing a set of putative correspondences;

iii) using the collection of matching points, estimating the homography between the background matter of the first and second digital images; and

iv) transforming the second digital image according to the estimated homography to align it with the first digital image.

9. The method of claim 7, wherein the step (c) of using the first digital image and aligned digital image to produce a subject mask comprises the steps of:

i) subtracting the aligned digital image from the first digital image to generate a difference image;

ii) processing the difference image to generate a mask image;

iii) applying the mask image to the first digital image to select an area in which the subject is located, whereby the selected subject area is used in the compositing of a new image with a different background.

10. The method of claim 7, wherein the step (a) of receiving two digital images comprises the steps of:

i) capturing and displaying the first digital image;

ii) displaying and, with user intervention, coarsely aligning a live image with the first digital image; and

iii) capturing the live image to yield the second digital image, wherein the second digital image is coarsely adjusted by the user to include substantially the same background matter without the subject.

11. The method of claim 7, wherein the step (a) of receiving the two digital images comprises the steps of:

i) initiating a composite image capture process in a digital camera by capturing and storing the first digital image and displaying a portion of the first digital image that is intended to overlap with the second digital image; and

ii) displaying a live image in an overlapping relationship with the first digital image to enable a user to coarsely align the live image with the first digital image before capturing the live image as the second digital image.

12. A system for producing a foreground mask from a digital image, said system comprising:

a digital camera for capturing two digital images, each having a plurality of pixels, wherein a first digital image is obtained from a first original scene image including a foreground photographed against a background having arbitrary and unknown content matter, and a second digital image is obtained from a second original scene image including substantially the same background but without the foreground; and

a processor for (a) automatically aligning the second digital image to the first digital image to generate an aligned digital image including the background without the foreground, wherein the automatic aligning is achieved without user intervention; and

(b) using the first digital image and the aligned digital image to produce a foreground mask relating to an area in the first digital image pertaining to the foreground, whereby the foreground mask is subsequently used to generate a composite digital image.

13. The system of claim 12, wherein the processor subtracts the aligned digital image from the first digital image to generate a difference image, processes the difference image to generate a mask image, and applies the mask image to the first digital image to select an area in which the foreground is located, whereby the selected foreground area is used in the compositing of a new image with a different background.

14. The system of claim 12, wherein the camera includes a view finder for displaying the digital images and a camera processor coupled to the view finder for (a) enabling a display on the view finder of a portion of the first digital image that is to overlap with the second digital image and (b) enabling display of a live image in an overlapping relationship with the first digital image on the view finder to enable a user to coarsely align the live image with the first digital image before capturing the live image as the second digital image.

15. The system as claimed in claim 12 wherein the processor includes an alignment algorithm for performing a homographic transformation, said algorithm:

a) choosing a set of points in the first digital image and locating their corresponding points in the second digital image using a similarity measure, thereby generating a collection of points;

b) for each prospective point in the collection, selecting a matching point in the second digital image based on a neighborhood similarity measure evaluated within an area centered on the prospective point in the collection, thereby generating a collection of matching points representing a set of putative correspondences;

c) using the collection of matching points, estimating the homography between the background matter of the first and second digital images; and

d) transforming the second digital image according to the estimated homography to align it with the first digital image.

16. A digital camera for capturing images that are subsequently useful in a compositing process, said camera comprising:

an imaging device for capturing image data;

a memory for storing the image data from the imaging device;

a digital processing section coupled to the imaging device and the memory to enable the capture and storage of a first digital image including a subject photographed against a background;

an electronic view finder coupled to the digital processing section for displaying a portion of the first digital image in a first display zone on the view finder; and

said digital processing section enabling the capture and display on the view finder of a second digital image including substantially the same background, but without the subject, in an overlapping relationship with the displayed portion of the first digital image, whereby the second digital image is displayed as a live image in a second display zone on the viewfinder to enable a user to coarsely align the live image with the first digital image before capturing the live image as the second digital image, thereby ensuring that the two images are captured from substantially the same viewpoint.

17. The camera claimed in claim 16 wherein the second display zone surrounds the first display zone in the view finder of the camera.

18. The camera claimed in claim 17 wherein the first display zone displays the subject substantially in the center of a display provided by the view finder and the second display zone is substantially around the periphery of the display, whereby the border separating the two zones generally separates the subject from the background for many typical capture situations.

19. The camera claimed in claim 16 wherein the first display zone surrounds the second display zone in the viewfinder of the camera.

20. The camera claimed in claim 16 wherein the first display zone displays the background portion of the first digital image substantially around the periphery of the display and the second display zone displays the background substantially in the center of the display, whereby the border separating the two zones generally separates the periphery of the saved background from the live central portion of the background.

21. The camera claimed in claim 16 wherein the first and second digital images are overlapped and combined at least in part in a display provided by the viewfinder in order to assist in coarse alignment of the live image with the first digital image.