Method and apparatus for background replacement in still photographs

• US 7,834,894 B2
• Filed: 04/03/2007
• Issued: 11/16/2010
• Est. Priority Date: 04/03/2007
• Status: Active Grant

• Current Assignee: Shutterfly, LLC
• Original Assignee: Lifetouch Incorporated (Shutterfly, LLC)
• Inventors: Adolphi, Earl John, Swanson, Richard Lee, Surma, Michael John, Reinert Nash, John Robert
• Primary Examiner(s): Yang; Ryan R
• Application Number:
  US11/732,507
• Publication Number:
  US 20080246777A1
• Time in Patent Office: 1,323 Days
• Field of Search: 345/629, 345/634
• US Class Current: 345/634
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A first digital image is acquired of a framed area while illuminating the background and foreground object under a first lighting condition. A second digital image is then acquired of the same framed area while illuminating the background and foreground object under a second lighting condition. Preferably, the first lighting condition illuminates the background without illuminating the foreground object so that a silhouette of the foreground object is acquired in the first image. The second lighting condition illuminates the foreground object (e.g., with frontal lights). Due to the difference in the illumination between the background and silhouette in the first image, an alpha mask can be created from the first acquired image. Using the mask, the background from the second image can be removed and replaced by virtually any other desired background image.

Claims

1. A method of replacing a background in an image with a different background, the method comprising:

acquiring a first image of the foreground object and the background under a first lighting condition in which the foreground object and the background are illuminated;

acquiring a second image of the foreground object and the background under a second lighting condition in which the background is illuminated;

computing a mixing function with at least one processor device based on the second image, the mixing function including a value for each pixel of the second image, the value for each pixel being substantially equal to a first value for pixels that correspond to the background, substantially equal to a second value for pixels that correspond to the foreground object, and between the first value and the second value for pixels that correspond to border regions between the foreground object and the background;

computing an object image function with the at least one processor device using the first image and the mixing function, the object image function including an image of the foreground object from the first image and having at least part of the background from the first image removed; and

generating a new image including the image of the foreground object from the first image and at least portions of the different background by mixing the different background with the object image function, wherein;

pixels of the new image that correspond to the background of the second image are substantially unchanged from pixels of the different background,

pixels of the new image that correspond to the foreground object of the second image are substantially replaced by pixels that correspond to the foreground object of the first image, and

pixels of the new image that correspond to the border regions between the foreground object and the background of the second image are mixed to include contributions from both the foreground object of the first image and the different background.

2. The method of claim 1, wherein the step of computing the mixing function includes computing an alpha value for alpha blending for each pixel of the first image.

3. The method of claim 1, wherein acquiring the second image under the second lighting condition comprises illuminating the background so that it is brighter than the foreground object.

4. The method of claim 1, wherein generating the new image generates a composite image in which the different background image appears to be behind the foreground object.

5. The method of claim 4, wherein:

computing a mixing function comprises computing an alpha value (α

) for alpha blending for each pixel of the second image;

computing the object image function comprises computing α

F^c for each pixel, where F^c denotes the color vector of the pixel of the first image of the foreground object with the background removed; and

generating the composite image comprises computing each pixel of the composite image according to the formula;

R=α

F^c+(1−

α

)B^new,

where R is the color vector of each pixel of the composite image and B^new is the color vector of the third image.

6. The method of claim 4, further comprising transmitting the object image function from a first location to a second location over a computer network, wherein generating the composite image comprises generating the compositing function based on the transmitted object image function.

7. The method of claim 1, further comprising determining a maximum acceptable movement speed of the foreground object and computing a time interval within which the first and second images should be acquired based on the maximum acceptable movement speed.

8. The method of claim 7, further comprising setting a maximum number of pixels by which the first and second images are permitted to be offset from each other, wherein computing the time interval comprises computing the time interval based on the number of pixels by which the first and second images are permitted to be offset from each other.

9. The method of claim 8, wherein computing the time interval comprises computing the time interval according to the formula

t=w_sd_t/(w_cv_t)

where v_t is the tolerated speed of the foreground object,

w_s is the width of the image at the foreground object,

d_t is the number of pixels by which the first and second images are permitted to be offset from each other,

w_c is the width (in pixels) of the first and second images, and

t is the time to capture both images.

10. The method of claim 8, wherein setting the maximum number pixels by which the first and second images are permitted to be offset from each other comprises limiting the offset to a fraction of a pixel.

11. The method of claim 7, wherein acquiring the first and second images comprises acquiring the first and second images under flash illumination and acquiring the second image after acquiring the first image.

12. The method of claim 1, wherein acquiring the first and second images comprises acquiring the first and second images under flash illumination and acquiring the second image within a time interval shorter than one (1) millisecond after acquiring the first image.

13. The method of claim 1, wherein the foreground object cannot move and wherein acquiring the first and second images comprises acquiring the first and second images under flash illumination and acquiring the second image at any given future time after acquiring the first image.

14. A method of computing an image from first and second images, the method comprising:

computing a mixing function with at least one processor device based on the second image, the second image being of the foreground object and the background under a second lighting condition in which the background is illuminated, the mixing function including a value for each pixel of the second image, the value for each pixel being substantially equal to a first value for pixels that correspond to the background, substantially equal to a second value for pixels that correspond to the foreground object, and between the first value and the second value for pixels that correspond to border regions between the foreground object and the background; and

computing an object image function with the at least one processor device using the first image and the mixing function, the first image being of a foreground object and the background under a first lighting condition in which the foreground object and the background are illuminated, the object image function including an image of the foreground object from the first image and having at least part of the background from the first image removed, wherein the object image function is arranged and configured to be mixed with a third image to generate a new image in which the third image appears to be behind the foreground object.

15. The method of claim 14, further comprising acquiring the second image with a digital camera, wherein the background is illuminated so that it is brighter than the foreground object under the first lighting condition.

16. The method claim 15, wherein the foreground object appears substantially black in the second image under the second lighting condition.

17. A method of imaging a foreground object placed in front of a background, the method comprising:

acquiring a first image of the foreground object and the background while illuminating the background under a first lighting condition relative to the foreground object; and

acquiring a second image of the foreground object and the background with a camera while illuminating the background under a second lighting condition relative to the foreground object within a predetermined time interval of acquiring the first image, the predetermined time interval being computed based at least in part on a maximum acceptable speed of movement of the foreground object and a maximum number of pixels by which the first and second images are permitted to be offset from each other, wherein the maximum number of pixels is less than one pixel.

18. The method of claim 17, wherein computing the predetermined time interval comprises computing the predetermined time interval according to the formula

t=w_sd_t/(w_cv_t)

where v_t is the tolerated speed of the foreground object,

w_s is the width of the image at the foreground object,

d_t is the number of pixels by which the first and second images are permitted to be offset from each other,

w_c is the width (in pixels) of the first and second images, and

t is the time to capture the two images.

19. The method of claim 17, wherein acquiring the first and second images comprises acquiring the first and second images under flash illumination and acquiring the second image after acquiring the first image.

20. An imaging system for imaging a scene having a foreground object placed in front of the background, comprising:

a camera system comprising an interline transfer sensor array comprising alternating photosensor elements and storage elements arranged and configured to store signals transferred from corresponding photosensor elements, the camera system arranged and configured to acquire an image of the scene; and

a lighting system configured to generate a first lighting condition, in which the background appears to the camera system brighter than the foreground object, and a second lighting condition, in which both the foreground object and the background are illuminated,

wherein the lighting system is arranged and configured to sequentially generate the first and second lighting conditions within a time interval computed based on a maximum acceptable movement speed of the foreground object, and

the camera system is arranged and configured to acquire a first image of the scene under the first lighting condition with the photosensor elements, and arranged and configured to acquire a second image of the scene under the second condition with the photosensor elements after transferring energy associated with the first image to the storage elements.

21. The imaging system of claim 20, wherein the lighting system is configured to generate the second lighting condition after the generating the first lighting condition within the time interval.

22. The imaging system of claim 21, wherein the time interval is less than one (1) millisecond.

23. The imaging system of claim 20, further comprising a mechanical shutter and a controller, wherein the controller is arranged and configured to control the mechanical shutter to open prior to capture of the first image, remain open while the camera system acquires the second image, and close after the camera system acquires the second image to block further exposure of the photosensor elements from ambient light.

24. The imaging system of claim 20, further comprising an image processor arranged and configured to receive the first and second image and

compute a mixing function based on the first image;

and

using the second image and the mixing function, computing an object image function relating to an image of the foreground object with a reduced background intensity relative to the foreground object as compared to the second image.

25. A storage medium having encoded thereon computer-readable instructions that, when executed by a computer, cause the computer to:

compute a mixing function based on the second image, the second image being of the foreground object and the background under a second lighting condition in which the background is illuminated, the mixing function including a value for each pixel of the second image, the value for each pixel being substantially equal to a first value for pixels that correspond to the background, substantially equal to a second value for pixels that correspond to the foreground object, and between the first value and the second value for pixels that correspond to border regions between the foreground object and the background; and

compute an object image function using the first image and the mixing function, the first image being of a foreground object and the background under a first lighting condition in which the foreground object and the background are illuminated, the object image function including an image of the foreground object from the first image and having at least part of the background from the first image removed, wherein the object image function is arranged and configured to be mixed with a third image to generate a new image in which the third image appears to be behind the foreground object.

26. A method of imaging a foreground object placed in front of a background, the method comprising:

determining a maximum acceptable movement speed of the foreground object and computing a time interval within which a first image and a second image should be acquired based on the maximum acceptable movement speed;

setting a maximum number of pixels by which the first and second images are permitted to be offset from each other, wherein computing the time interval comprises computing the time interval with a processor device based on the number of pixels by which the first and second images are permitted to be offset from each other according to the formula

t=w_sd_t/(w_cv_t)

where v_t is the tolerated speed of the foreground object,

w_s is the width of the image at the foreground object,

d_t is the number of pixels by which the first and second images are permitted to be offset from each other,

w_c is the width (in pixels) of the first and second images, and

t is the time to capture both images;

acquiring the first image of the foreground object and the background while illuminating the background under a first lighting condition relative to the object;

acquiring the second image of the foreground object and the background while illuminating the background under a second lighting condition relative to the object;

computing a mixing function based on a selected one of the images; and

using the image that was not selected and the mixing function, computing an object image function relating to an image of the foreground object with a reduced background intensity relative to the foreground object as compared to the image that was not selected.

27. A method of imaging an object placed in front of a background, the method comprising:

acquiring a first image of the foreground object and the background while illuminating the background under a first lighting condition relative to the object;

acquiring a second image of the foreground object and the background while illuminating the background under a second lighting condition relative to the object within a predetermined time interval of acquiring the first image, the predetermined time interval being computed based on a maximum acceptable speed of movement of the foreground object; and

setting a maximum number pixels by which the first and second images are permitted to be offset from each other, wherein computing the predetermined time interval comprises computing the predetermined time interval with a processor device based on the maximum number of pixels by which the first and second images are permitted to be offset from each other, wherein computing the predetermined time interval further comprises computing the predetermined time interval according to the formula

t=w_sd_t/(wv_t)

where v_t is the tolerated speed of the foreground object,

w_s is the width of the image at the foreground object,

d_t is the number of pixels by which the first and second images are permitted to be offset from each other,

w_c is the width (in pixels) of the first and second images, and

t is the time to capture the two images.