Image extraction from complex scenes in digital video

• US 7,003,061 B2
• Filed: 12/21/2000
• Issued: 02/21/2006
• Est. Priority Date: 12/21/2000
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wilensky, Gregg D.
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US09/747,945
• Publication Number:
  US 20020118875A1
• Time in Patent Office: 1,888 Days
• Field of Search: 348/239, 348/240, 348/241, 348/242, 348/243, 348/446, 348/135, 375/360, 382/283, 382/154, 382/194, 382/146, 382/199, 382/256, 382/266, 382/284, 382/282, 345/591
• US Class Current: 375/360
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20041 Distance transform
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

The invention provides methods and apparatus, including computer program products, implementing and using techniques for masking and extracting a foreground portion from a background portion of a digital image. In the method, a first input defining a first border region is received, which includes at least a part of the foreground portion and at least a part of the background portion in a first digital image. A second input defining a second border region is received, which includes at least a part of the foreground portion and at least a part of the background portion in a second digital image. An intermediary border region is interpolated for an image intermediary in time to the first and second digital images and the first, second, and intermediary border regions are used for masking the foreground portion from the background portion in the digital video.

Claims

1. A method for masking a foreground portion from a background portion of a digital video, the method comprising:

receiving a first input defining a first border region that includes a border between a foreground portion and a background portion of a first digital image, the first digital image being one of a sequence of digital images defining a digital video, the first border region further including only a part of the foreground portion and only a part of the background portion, the border consisting of pixels that include contributions from the foreground portion and the background portion;

receiving a second input defining a second border region that includes the border in a second digital image, the second digital image being another one of the sequence of digital images, the second border region further including only a part of the foreground portion and only a part of the background portion;

interpolating between the first and second border regions to define an intermediary border region for an image intermediary in the sequence to the first and second digital images; and

using the first, second, and intermediary border regions for masking the foreground portion from the background portion in the digital video.

2. The method of claim 1 wherein receiving a first and a second input comprise:

receiving user inputs defining the border regions; and

where interpolating an intermediary border region comprises;

interpolating an intermediary border region automatically without user input.

3. The method of claim 1 wherein using the first, second, and intermediary border regions for masking the foreground portion from the background portion comprises:

determining, for a pixel in a border region, whether the pixel includes data that is associated with the foreground portion; and

using the result of the determining step to mask the foreground portion from the background portion in the digital video.

4. The method of claim 1, further comprising the steps of:

estimating an intrinsic color value for a pixel in the first, second, and intermediary border regions; and

using the estimated intrinsic color value for extracting the foreground position from the background portion.

5. The method of claim 1 wherein receiving inputs indicating the border regions comprises:

generating an inside path located inside a foreground portion; and

generating an outside path located outside the foreground portion and enclosing the inside path,

wherein pixels between the inside and outside paths belong to a border region.

6. The method of claim 5 wherein generating an inside path and an outside path comprise:

generating a vector-based inside path and a vector-based outside path.

7. The method of claim 1 wherein receiving inputs indicating the border regions comprises:

generating a single path having a width encompassing that of the border region.

8. The method of claim 7 wherein generating a single path comprises:

generating a vector-based single path.

9. The method of claim 7 wherein generating a single path comprises:

generating a single path having variable thickness throughout its length.

10. The method of claim 1 wherein using the first, second, and intermediary border regions for masking the foreground portion from the background portion comprises:

masking the foreground portion and a portion of the border region; and

applying Gaussian blur to the portion of the border region.

11. The method of claim 10, further comprising displaying the masked foreground portion and the portion of the border region.

12. The method of claim 1 wherein using the first, second, and intermediary border regions for masking the foreground portion from the background portion comprises:

masking the foreground portion and a portion of the border region;

determining the opacities of the pixels in the portion of the border region that is displayed with the foreground portion; and

changing the opacities for pixels having opacities greater than a threshold value.

13. The method of claim 12, further comprising displaying the masked foreground portion and the portion of the border region.

14. The method of claim 1, wherein the border is a first border, the method further comprising:

receiving a third input defining a third border region tat includes a second border that is between a second foreground portion and the background portion in the first digital image, the third border region further including only a part of the second foreground portion and only a part of the background portion;

receiving a fourth input defining a fourth border region that includes the second border in the second digital image, the fourth border region further including only a part of the second foreground portion and only a part of the background portion; and

interpolating between the third and fourth border regions to define an intermediary border region for the second foreground portion for the intermediary digital image.

15. The method of claim 1 further comprising:

receiving an input defining a first internal border region, the first internal border region being enclosed by the first border region, in the first digital image;

receiving an input defining a second internal border region, the second internal border region being enclosed by the second border region, in the second digital image;

interpolating an intermediary internal border region for the image intermediary in time to the first and second digital images; and

using the first, second, and intermediary border regions, and the first internal, second internal, and intermediary internal border regions for masking the foreground portion from the background portion in the digital video.

16. The method of claim 15, wherein the first internal border region, the second internal border region, and the intermediary internal border region comprise at least a part of the foreground portion.

17. The method of claim 15, wherein the first internal border region, the second internal border region, and the intermediary internal border region comprise at least a part of the background portion.

18. The method of claim 15, wherein the first internal border region, the second internal border region, and the intermediary internal border region comprise at least a part of the foreground portion and at least a part of the background portion.

19. The method of claim 1, wherein the foreground portion is an object.

20. A method for masking an object from a background portion in a digital video, the method comprising:

receiving a first user input defining a first border region that includes a border between an object and a background portion of a first digital image, the first digital image being one of a sequence of digital images that define a digital video, the first border region further including only a part of the object and only a part of the background portion, the border consisting pixels that include contributions from the object and the background portion;

receiving a second user input defining a second border region that includes the border in a second digital image, the second digital image being another one of the sequence of digital images, the second border region further including only a part of the object and only a part of the background portion;

interpolating between the first and second border regions to define an intermediary border region for an image intermediary in the sequence to the first and second digital images;

using the first, second, and intermediary border regions for masking the object from the background portion in the digital video;

using the first, second, and intermediary border regions for extracting the object from the background portion in the digital video; and

placing the masked and extracted object against a new background in another digital video.

21. Computer software tangibly embodied in a computer-readable medium or a propagated carrier signal, for masking a foreground portion of from a background portion of a digital video, the software comprising instructions to perform the following operations:

receive a first input defining a first border region that includes a border between a foreground portion and a background portion of a first digital image, the first digital image being one of a sequence of digital images that define a digital video, the first border region further including only a part of the foreground portion and only a part of the background portion, the border consisting of pixels that include contributions from the foreground portion and the background portion;

receive a second input defining a second border region that includes the border in a second digital image, the second digital image being another one of the sequence of digital images, the second border region further including only a part of the foreground portion and only a part of the background portion;

interpolate between the first and second border regions to define an intermediary border region for an image intermediary in the sequence to the first and second digital images; and

use the first, second, and intermediary border regions for masking the foreground portion from the background portion in the digital video.

22. The computer software of claim 21 wherein the instructions to receive a first and a second input comprise instructions to:

receive user inputs defining the border regions; and

where interpolating an intermediary border region comprises;

interpolate an intermediary border region automatically without user input.

23. The computer software of claim 21 wherein the instructions to use the first, second, and intermediary border regions for masking the foreground portion from the background portion comprises instructions to:

determine, for a pixel in a border region, whether the pixel includes data that is associated with the foreground portion; and

use the result of the determining step to mask the foreground portion from the background portion in the digital video.

24. The computer software of claim 21, further comprising instructions to perform the following operations:

estimate an intrinsic color value for a pixel in the first, second, and intermediary border regions; and

use the estimated intrinsic color value for extracting the foreground portion from the background portion.

25. The computer software of claim 21 wherein the instructions to receive inputs indicating the border regions comprise instructions to:

generate an inside path located inside a foreground portion; and

generate an outside path located outside the foreground portion and enclosing the inside path,

wherein pixels between the inside and outside paths belong to a border region.

26. The computer software of claim 25 wherein the instructions to generate an inside path and an outside path comprise instructions to generate a vector-based inside path and a vector-based outside path.

27. The computer software of claim 22 wherein the instructions to receive inputs indicating the border regions comprise instructions to generate a single path having a width encompassing that of the border region.

28. The computer software of claim 27 wherein the instructions to generate a single pat comprise instructions to generate a vector-based single path.

29. The computer software of claim 27 wherein the instructions to generate a single path comprise instructions to generate a single path having variable thickness throughout its length.

30. The computer software of claim 21 wherein the instructions to use the first, second, and intermediary border regions for masking the foreground portion from the background portion comprise instructions to:

mask the foreground portion and a portion of the border region; and

apply Gaussian blur to the portion of the border region.

31. The computer software of claim 30, further comprising instructions to display the masked foreground portion and the portion of the border region.

32. The computer software of claim 21 wherein the instructions to use the first, second, and intermediary border regions for masking the foreground portion from the background portion comprise instructions to:

mask the foreground portion and a portion of the border region;

determine the opacities of the pixels in the portion of the border region that is displayed with the foreground portion; and

change the opacities for pixels having opacities greater than a threshold value.

33. The computer software of claim 32, further comprising instructions to display the masked foreground portion and the portion of the border region.

34. The computer software of claim 21, wherein the border is a first border, the computer software further comprising instructions to perform the following operations:

receive a third input defining a third border region that includes a second border that is between a second foreground portion and the background portion in the first digital image, the third border region further including only a part of the second foreground portion and only a part of the background portion;

receive a fourth input defining a fourth border region that includes the second border in the second digital image, the fourth border region further including only a part of the second foreground portion and only a part of the background portion; and

interpolate between the third and fourth border regions to define an intermediary border region for the second foreground portion for the intermediary digital image.

35. The computer software of claim 21 further comprising instructions to perform the following operations:

receive an input defining a first internal border region, the first internal border region being enclosed byte first border region, in the first digital image;

receive an input defining a second internal border region, the second internal border region being enclosed by the second border region, in the second digital image;

interpolate an intermediary internal border region for the image intermediary in time to the first and second digital images; and

use the first, second, and intermediary border regions, and the first internal, second internal, and intermediary internal border regions for masking the foreground portion from the background portion in the digital video.

36. The computer software of claim 35, wherein the first internal border region, the second internal border region, and the intermediary internal border region comprise at least a part of the foreground portion.

37. The computer software of claim 35, wherein the first internal border region, the second internal border region, and the intermediary internal border region comprise at least a part of the background portion.

38. The computer software of claim 35, wherein the first internal border region, the second internal border region, and the intermediary internal border region comprise at least a part of the foreground portion and at least a part of the background portion.

39. The computer software of claim 21, wherein the foreground portion is an object.

40. Computer software, tangibly embodied in a computer-readable medium or a propagated carrier signal, for masking an object from a background portion in a digital video, the software comprising instructions to perform the following operations:

receive a first user input defining a first border region that includes a border between an object and a background portion of a first digital image, the first digital image being one of a sequence of digital images that define a digital video, the first border region further including only a part of the object and only a part of the background portion, the border consisting of pixels that include contributions from the object and the background portion;

receive a second user input defining a second border region that includes the border in a second digital image, the second digital image being another one of the sequence of digital images, the second border region further including only a part of the object and only a part of the background portion;

interpolate between the first and second border regions to define an intermediary border region for an image intermediary in the sequence to the first and second digital images;

use the first, second, and intermediary border regions for masking the object from the background in the digital video;

use the first, second, and intermediary border regions for extracting the object from the background portion in the digital video; and

place the masked and extracted object against a new background in another digital video.