Background removal in a live video

• US 7,720,283 B2
• Filed: 08/31/2006
• Issued: 05/18/2010
• Est. Priority Date: 12/09/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sun, Jian, Shum, Heung-Yeung, Tang, Xiaoou, Zhang, Weiwei
• Primary Examiner(s): Couso; Jose L
• Application Number:
  US11/469,371
• Publication Number:
  US 20070133880A1
• Time in Patent Office: 1,356 Days
• Field of Search: 382/173, 382/176, 382218-222, 382/254, 382/270, 382/274, 358/3.27, 358464-466, 348/251, 348/603, 348/619
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

Exemplary systems and methods segment a foreground from a background image in a video sequence. In one implementation, a system refines a segmentation boundary between the foreground and the background image by attenuating background contrast while preserving contrast of the segmentation boundary itself, providing an accurate background cut of live video in real time. A substitute background may then be merged with the segmented foreground within the live video. The system can apply an adaptive background color mixture model to improve segmentation of foreground from background under various background changes, such as camera movement, illumination change, and movement of small objects in the background.

Claims

1. A method of segmenting a foreground from a background image, comprising:

hardware for determining color and contrast attributes of pixels of the background image to be used in a video sequence, wherein the background image occupies an entire area of each frame of the video sequence except when the foreground hides part of the background image;

hardware for approximating a segmentation boundary demarcating the foreground based on differences between color and contrast attributes of pixels of the foreground versus the color and contrast attributes of pixels of the background image; and

hardware for refining the segmentation boundary between the foreground and the background image by adaptively attenuating a background contrast, wherein the attenuating includes decreasing a value of the contrast attribute of pixel pairs in the background image while maintaining a value of the contrast attribute of pixel pairs associated with the segmentation boundary.

2. The method as recited in claim 1, wherein the refining associates a pixel pair with the segmentation boundary when a value of the contrast attribute of the pixel pair is dissimilar to a value of the contrast attribute of a corresponding pixel pair in the background image.

3. The method as recited in claim 1, wherein the background image is stationary through the video sequence.

4. The method as recited in claim 1, further comprising obtaining the video sequence from a single camera.

5. The method as recited in claim 1, wherein the contrast attributes of pixels of the foreground and background image comprise a color image gradient.

6. The method as recited in claim 1, further comprising segmenting the foreground from the background image at the segmentation boundary.

7. The method as recited in claim 6, further comprising merging the foreground with a different background image.

8. The method as recited in claim 1, further comprising applying an adaptive mixture model to improve segmenting the foreground from the background image under various background changes.

9. The method as recited in claim 8, further including maintaining the background image during the video sequence, wherein the maintaining includes compensating for a change in the background image during the video sequence by adaptively applying a weighted mixture of a per-pixel background color model and a global background color model.

10. The method as recited in claim 9, further comprising learning the global background color model from the background image.

11. The method as recited in claim 9, further comprising learning a per-pixel variance of the per-pixel background color model from a background initiation phase.

12. The method as recited in claim 9, further comprising weighting the mixture of the per-pixel background color model and the global background color model according to the type of the change in the background image.

13. The method as recited in claim 9, wherein the type of change in the background image comprises one of:

noise, illuminance change, small movement of the background due to camera motion, a small object moving across the background, and a small object entering the background and coming to rest in the background.

14. A system for segmenting a foreground from a background of a video sequence, comprising:

hardware comprising a foreground extractor to separate the foreground from the background of the video sequence, the foregoing extractor comprising;

a segmentation engine to label pixels of the video sequence as either foreground or background;

a boundary determination engine to calculate a boundary between the foreground and the background;

a color evaluator to calculate the boundary according to foreground colors and background colors; and

a contrast evaluator to calculate the boundary according to foreground contrasts and background contrasts, the contrast evaluator comprises a background contrast attenuator to attenuate contrast in the background while adaptively preserving contrast associated with the boundary.

15. The system as recited in claim 14, further comprising a background image manager to obtain and track color and contrast attributes of an image to be used as the background of the video sequence.

16. The system as recited in claim 14, further comprising a single camera to obtain the video sequence.

17. The system as recited in claim 14, further comprising a background maintenance engine to compensate for a change in the background image during the video sequence by adaptively applying a weighted mixture of a per-pixel background color model and a global background color model.

18. The system as recited in claim 17, wherein the background maintenance engine learns the global background color model from the background and the background maintenance engine learns the per-pixel background color model from a background initiation.

19. The system as recited in claim 14, further comprising a background substitution engine to merge a different background with the separated foreground in the video sequence.

20. A computer-implemented system for segmenting a foreground from a background image, comprising:

hardware means for segmenting the foreground from the background image in a video sequence;

hardware means for determining a segmentation boundary between the foreground and the background image;

hardware means for selecting pixel pairs associated with the segmentation boundary according to a difference in contrast with corresponding pixel pairs in the background image; and

hardware means for refining the segmentation boundary by attenuating background contrast while preserving contrast of the pixel pairs associated with the segmentation boundary.