Image extraction from complex scenes in digital video

• US 20060126719A1
• Filed: 01/12/2006
• Published: 06/15/2006
• Est. Priority Date: 12/21/2000
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wilensky, Gregg D.
• Granted Patent: US 7,729,422 B2
• Time in Patent Office: Days
• US Class Current: 375/240.10
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20041 Distance transform
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

The invention provides methods and apparatus, including computer program products, implementing and using techniques for masking and extracting a foreground portion from a background portion of a digital image. In the method, a first input defining a first border region is received, which includes at least a part of the foreground portion and at least a part of the background portion in a first digital image. A second input defining a second border region is received, which includes at least a part of the foreground portion and at least a part of the background portion in a second digital image. An intermediary border region is interpolated for an image intermediary in time to the first and second digital images and the first, second, and intermediary border regions are used for masking the foreground portion from the background portion in the digital video.

Claims

1. A method for masking a foreground portion from a background portion of a digital video, the method comprising:

receiving an input defining an initial border region that includes a border between a foreground portion and a background portion of an initial digital image, the initial digital image being one of a sequence of digital images that define a digital video, the initial border region further including only a part of the foreground portion and only a part of the background portion, the border consisting of pixels that include contributions from the foreground portion and the background portion;

automatically generating, based on the initial border region, a border region for another digital image of the sequence of digital images, the automatically generated border region including the border in the other digital image, only a part of the foreground portion, and only a part of the background portion; and

using the initial and automatically generated border regions for masking the foreground portion from the background portion in the digital video.

2. The method of claim 1, wherein receiving an input comprises receiving a user input defining the border region.

3. The method of claim 1, wherein using the initial and automatically determined border regions for masking the foreground portion from the background portion comprises:

determining, for a pixel in a border region, whether the pixel comprises data that is associated with the foreground portion; and

using the result of the determining step to mask the foreground portion from the background portion in the digital video.

4. The method of claim 1, further comprising:

estimating an intrinsic color value for a pixel in the initial border region; and

using the estimated intrinsic color value for extracting the foreground portion from the background portion.

5. The method of claim 2, wherein receiving a user input indicating the border regions comprises one of:

generating an inside path located inside the foreground portion, and generating an outside path located outside the foreground portion and enclosing the inside path, wherein pixels between the inside and outside paths belong to a border region;

generating a vector-based inside path and a vector-based outside path;

or

generating a single path having a width encompassing the border region.

6. The method of claim 1, wherein using the initial and the automatically generated border regions for masking the foreground portion from the background portion comprises:

masking the foreground portion and a portion of the border region; and

applying Gaussian blur to the portion of the border region.

7. The method of claim 6, further comprising displaying the masked foreground portion and the portion of the border region.

8. The method of claim 1, wherein using the initial and the automatically generated border regions for masking the foreground portion from the background portion comprises:

masking the foreground portion and a portion of the border region;

determining the opacities of the pixels in the portion of the border region that is displayed with the foreground portion; and

changing the opacities for pixels having opacities greater than a threshold value.

9. The method of claim 1, wherein automatically generating a border region for another digital image comprises:

identifying an edge path for the initial border region, the edge path indicating edges of the foreground portion in the initial digital image;

identifying an edge path for the automatically generated border region, the edge path indicating edges of the foreground portion in the other digital image;

determining an edge path transformation between the initial digital image and the other digital image; and

applying the edge path transformation to the inside path and the outside path in the initial digital image in order to generate a border region in the other digital image.

10. The method of claim 9, wherein identifying the edge path for the other automatically generated border region comprises:

identifying the edge path using a snake algorithm operating on a previously identified edge path.

11. The method of claim 9, wherein identifying an edge path for the initial border region comprises:

finding points in the border region that have image gradient values exceeding a threshold value.

12. The method of claim 11, further comprising:

calculating an image gradient value for each color channel in the initial border region.

13. The method of claim 9, wherein identifying an edge path for the initial border region comprises:

copying the inside path and the outside path;

moving the copies of the inside path and the outside path towards each other and towards edges of the foreground portion using a snake algorithm; and

identifying the edge path as the path where copy of the inside path and the copy of the outside path converge into one path.

14. The method of claim 9, wherein determining an edge path transformation comprises:

determining a set of local edge path transformations, the set of local edge path transformations together forming the edge path transformation.

15. The method of claim 14, wherein determining a set of local edge path transformations comprises:

determining a local edge path transformation for each point in the edge path.

16. The method of claim 14, wherein determining a set of local edge path transformations comprises:

determining a translation, a rotation, and a scaling for each local edge path transformation in the set of local edge path transformations.

17. The method of claim 9, wherein applying the edge path transformation to the inside and outside path comprises:

applying a set of local transformations to each point in the inside path and to each point in the outside path, the set of local transformations forming the edge path transformation.

18. A computer program product, tangibly stored a machine-readable medium, for masking a foreground portion from a background portion of a digital video, the product comprising instructions operable to cause a processor to perform a method comprising:

receiving an input defining an initial border region that includes a border between a foreground portion and a background portion of an initial digital image, the initial digital image being one of a sequence of digital images that define a digital video, the initial border region further including only a part of the foreground portion and only a part of the background portion, the border consisting of pixels that include contributions from the foreground portion and the background portion;

automatically generating, based on the initial border region, a border region for another digital image of the sequence of digital images, the automatically generated border region including the border in the other digital image, only a part of the foreground portion, and only a part of the background portion; and

using the initial and automatically generated border regions for masking the foreground portion from the background portion in the digital video.

19. The product of claim 18, wherein receiving an input comprises receiving a user input defining the border region.

20. The product of claim 18, wherein using the initial and automatically determined border regions for masking the foreground portion from the background portion comprises:

determining, for a pixel in a border region, whether the pixel comprises data that is associated with the foreground portion; and

using the result of the determining step to mask the foreground portion from the background portion in the digital video.