Object matting using flash and no-flash images

• US 20070263119A1
• Filed: 05/15/2006
• Published: 11/15/2007
• Est. Priority Date: 05/15/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sun, Jian, Shum, Heung-Yeung, Li, Yin, Kang, Sing
• Granted Patent: US 7,724,952 B2
• Time in Patent Office: Days
• US Class Current: 348/371
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 5/275 Generation of keying signals

Abstract

Foreground object matting uses flash/no-flash images pairs to obtain a flash-only image. A trimap is obtained from the flash-only image. A joint Bayesian algorithm uses the flash-only image, the trimap and one of the image of the scene taken without the flash or the image of the scene taken with the flash to generate a high quality matte that can be used to extract the foreground from the background.

Claims

1. A method of obtaining a matte suitable for separating a foreground object from a background object, the method comprising:

obtaining a flash-only image of a scene based on an image of the scene taken with a flash and an image of the scene taken without a flash;

obtaining a trimap based on the flash-only image; and

obtaining a matte suitable for separating a foreground object from a background object of the scene based on the trimap, the flash-only image and at least one of the image of the scene taken without the flash and the image of the scene taken with the flash.

2. The method of claim 1 wherein obtaining the matte includes using Bayesian matting of the flash-only image and Bayesian matting at least one of the image of the scene taken without the flash and the image of the scene taken with the flash.

3. The method of claim 1 wherein obtaining the matte comprises maximizing unknown variables {α

,F,B,F′

} in a log likelihood function L(α

,F,B,F′

|I,I′

) where α

is related to the matte, F is related to the foreground and B is related to the background for the image I comprising one of the image of the scene taken without the flash and the image of the scene taken with the flash, and F′

is the foreground of the flash-only image I′

.

4. The method of claim 3 wherein obtaining the matte comprises using partial derivatives of the log likelihood function L(α

,F,B,F′

|I,I′

).

5. The method of claim 4 wherein obtaining the matte comprises iteratively estimate the matte using the partial derivatives until changes between two successive iterations is below a selected threshold.

6. The method of claim 1 wherein obtaining the matte comprises selectively using information based on the flash-only image and at least one of the image of the scene taken without the flash and the image of the scene taken with the flash.

7. The method of claim 6 wherein obtaining the matte includes using Bayesian matting of the flash-only image and Bayesian matting of at least one of the image of the scene taken without the flash and the image of the scene taken with the flash.

8. The method of claim 7 wherein using Bayesian matting of the images comprises representing statistics of colors of the foregrounds and the backgrounds in the images as oriented Gaussian distributions.

9. The method of claim 1 wherein the flash is of the foreground.

10. The method of claim 1 wherein the flash is of the background.

11. The method of claim 1 wherein obtaining the trimap comprises:

using a first threshold to obtain a foreground region of high confidence;

using a second threshold) to obtain a plurality of foreground regions of low confidence; and

keeping only foreground regions of low confidence that overlap with the foreground region of high confidence.

12. A computer-readable medium having computer-executable instructions that when executed by a computer perform steps to process images comprising:

obtaining a flash-only image of a scene based on an image of the scene taken with a flash and at least one of an image of the scene taken without a flash and the image of the scene taken with the flash;

obtaining a matte suitable for separating a foreground object from a background object of the scene based on the flash-only image and at least one of the image of the scene taken without the flash and the image of the scene taken with the flash;

inserting the foreground object into another image having a different background object, using the matte and foreground color.

13. The computer readable medium of claim 12 wherein the instructions further comprise obtaining a trimap based on the flash-only image, and wherein obtaining the matte comprises obtaining the matte based on the trimap, the trimap comprising regions of classification as definitely foreground object, definitely background object and unknown.

14. The computer readable medium of claim 13 wherein obtaining the trimap includes ascertaining a region of high confidence of the foreground object and selectively ascertaining regions of low confidence of the foreground object that overlap with the region of high confidence of the foreground object.

15. The computer readable medium of claim 14 wherein obtaining the trimap includes selectively allowing the user to adjust classifications of regions of the trimap.

16. The computer readable medium of claim 13 wherein obtaining the matte comprises maximizing unknown variables {α

,F,B,F′

} in a log likelihood function L(α

,F,B,F′

|I,I′

) for each pixel classified as unknown, where a is related to the matte, F is related to the foreground and B is related to the background for the image I comprising one of the image of the scene taken without the flash and the image of the scene taken with the flash, and F′

is the foreground of the flash-only image I′

.

17. The computer readable medium of claim 16 wherein obtaining the matte comprises using partial derivatives of the log likelihood function L(α

,F,B,F′

|I,I′

).

18. The computer readable medium of claim 17 wherein obtaining the matte comprises iteratively estimating the matte using the partial derivatives until changes between two successive iterations is below a selected threshold.

19. A system for image processing comprising:

an image difference module adapted to provide a flash-only image of a scene based on an image of the scene taken with a flash and an image of the scene taken without a flash;

a trimap generation module adapted to generate a trimap based on the flash-only image, wherein the trimap comprises regions of classification as definitely foreground object, definitely background object and unknown;

a generation module adapted to generate a matte and foreground color suitable for separating a foreground object from a background object of the scene based on the trimap and information based on Bayesian matting of the flash-only image and Bayesian matting at least one of the image of the scene taken without the flash and the image of the scene taken with the flash; and

an image combining module adapted to use the matte and foreground color to separate the foreground object of the image from the background object of the image in order to insert the foreground object into another image having a different background object.

20. The system of claim 19 wherein the generation module is adapted to maximize unknown variables {α

,F,B,F′

} in a log likelihood function L(α

,F,B,F′

|I,I′

) for each pixel in of the trimap classified as unknown where a is related to the matte, F is related to the foreground and B is related to the background for the image I comprising one of the image of the scene taken without the flash and the image of the scene taken with the flash, and F′

is the foreground of the flash-only image I′

.