Image segmentation of foreground from background layers

• US 7,676,081 B2
• Filed: 10/17/2005
• Issued: 03/09/2010
• Est. Priority Date: 06/17/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Criminisi, Antonio, Cross, Geoffrey, Kolmogorov, Vladimir, Blake, Andrew
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): YEH, EUENG NAN
• Application Number:
  US11/252,017
• Publication Number:
  US 20060285747A1
• Time in Patent Office: 1,604 Days
• Field of Search: 382/164, 382/180, 358/538
• US Class Current: 382/164
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

Segmentation of foreground from background layers in an image may be provided by a segmentation process which may be based on one or more factors including motion, color, contrast, and the like. Color, motion, and optionally contrast information may be probabilistically fused to infer foreground and/or background layers accurately and efficiently. A likelihood of motion vs. non-motion may be automatically learned from training data and then fused with a contrast-sensitive color model. Segmentation may then be solved efficiently by an optimization algorithm such as a graph cut.

Claims

1. One or more computer readable storage media containing computer readable instructions that, when implemented, perform a method comprising:

determining a first likelihood ratio of motion as compared to non-motion for each of a first plurality of pixels of a first input image;

determining a first segment indicator for each of the first plurality of pixels, each first segment indicator indicating a foreground pixel or a background pixel;

determining a first likelihood ratio of color for each of the first plurality of pixels based on the first segment indicators;

determining a second likelihood ratio of motion as compared to non-motion for each of the second plurality of pixels of a second input image subsequent to the first input image;

determining a second segment indicator for each of the second plurality of pixels based on the second likelihood ratio of motion and the first likelihood ratio of color, each second segment indicator indicating a foreground pixel or a background pixel; and

storing the second segment indicators in a data store.

2. The one or more computer readable storage media of claim 1, further comprising determining a spatial smoothness for each of the second plurality of pixels based on a contrast model, and wherein determining the second segment indicators is based on the spatial smoothness.

3. The one or more computer readable storage media of claim 1, wherein determining the second segment indicators includes minimizing an energy function based on the second likelihood ratio of motion and the first likelihood ratio of color.

4. The one or more computer readable storage media of claim 3, wherein minimizing includes using a graph cut.

5. The one or more computer readable storage media of claim 4, further comprising modifying, based on the second segment indicators, a display of at least a portion of the second input image.

6. The one or more computer readable storage media of claim 4, further comprising determining one or more motion parameters of the first and second likelihood ratios of motion based on a set of segmented training images different from the first and second input images.

7. A computing environment comprising a processing unit configured to execute operations, the operations comprising:

determining a first likelihood ratio of motion as compared to non-motion for each of a first plurality of pixels in a first image in a first data store;

determining a first segment indicator for each of the first plurality of pixels, each first segment indicator indicating a foreground pixel or a background pixel;

determining a first likelihood ratio of color for each of the first plurality of pixels based on the first segment indicators;

determining a second likelihood ratio of motion as compared to non-motion for each of a second plurality of pixels, the second plurality of pixels contained in a second image;

determining a second segment indicator for each of the second plurality of pixels based on the second likelihood ratio of motion and the first likelihood ratio of color, each second segment indicator indicating a foreground pixel or a background pixel; and

storing the second segment indicators in a second data store.

8. The computing environment of claim 7, the operations further comprising determining a spatial smoothness for each of the second plurality of pixels based on a contrast model, and wherein determining the second segment indicators is based on the spatial smoothness.

9. The computing environment of claim 7, wherein determining the second segment indicators includes minimizing an energy function based on the second likelihood ratio of motion and the first likelihood ratio of color.

10. The computing environment of claim 9, wherein minimizing includes using a graph cut.

11. The computing environment of claim 10, the operations further comprising modifying, based on the second segment indicators, a display of at least a portion of the second input image.

12. The computing environment of claim 10, the operations further comprising determining one or more motion parameters of the first and second likelihood ratios of motion based on a set of segmented training images different from the first and second input images.

13. A method comprising:

determining, using a processing unit, a first likelihood ratio of motion as compared to non-motion for each of a first plurality of pixels, a first image comprising the first plurality of pixels, a first data store comprising the first image;

determining a first segment indicator for each of the first plurality of pixels, each first segment indicator indicating a foreground pixel or a background pixel;

determining a first likelihood ratio of color for each of the first plurality of pixels based on the first segment indicators;

determining a second likelihood ratio of motion as compared to non-motion for each of a second plurality of pixels, the second plurality of pixels contained in a second image;

determining a second segment indicator for each of the second plurality of pixels based on the second likelihood ratio of motion and the first likelihood ratio of color, each second segment indicator indicating a foreground pixel or a background pixel; and

storing the second segment indicators in a second data store.

14. The method of claim 13, further comprising determining a spatial smoothness for each of the second plurality of pixels based on a contrast model, and wherein determining the second segment indicators is based on the spatial smoothness.

15. The method of claim 13, wherein determining the second segment indicators includes minimizing an energy function based on the second likelihood ratio of motion and the first likelihood ratio of color.

16. The method of claim 15, wherein minimizing includes using a graph cut.

17. The method of claim 16, further comprising modifying, based on the second segment indicators, a display of at least a portion of the second input image.

18. The method of claim 16, further comprising determining one or more motion parameters of the first and second likelihood ratios of motion based on a set of segmented training images different from the first and second input images.

19. The method of claim 13, wherein the first data store is the same as the second data store.