Image segmentation of foreground from background layers

• US 8,103,093 B2
• Filed: 01/19/2010
• Issued: 01/24/2012
• Est. Priority Date: 06/17/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Blake, Andrew, Criminisi, Antonio, Cross, Geoffrey, Kolmogorov, Vladimir
• Primary Examiner(s): Bali, Vikkram
• Assistant Examiner(s): YEH, EUENG NAN
• Application Number:
  US12/689,246
• Publication Number:
  US 20100119147A1
• Time in Patent Office: 735 Days
• Field of Search: 382/164, 382/173, 358/538
• US Class Current: 382/164
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

Segmentation of foreground from background layers in an image may be provided by a segmentation process which may be based on one or more factors including motion, color, contrast, and the like. Color, motion, and optionally contrast information may be probabilistically fused to infer foreground and/or background layers accurately and efficiently. A likelihood of motion vs. non-motion may be automatically learned from training data and then fused with a contrast-sensitive color model. Segmentation may then be solved efficiently by an optimization algorithm such as a graph cut. Motion events in image sequences may be detected without explicit velocity computation.

Claims

1. A computer readable storage media containing computer readable instructions that, when executed by a computing device, cause the computing device to perform:

determining one or more motion parameters of a motion model by fitting the motion model to a distribution of temporal derivatives or spatial gradients of training pixels from a set of training images, individual training pixels from the set of training images having previously been labeled as foreground or background training pixels;

determining a motion likelihood for input pixels of an input image, the motion likelihood being determined using the one or more motion parameters of the motion model without a velocity of the input pixels;

determining a color likelihood for segmenting the input pixels of the input image; and

automatically generating segmentation indicators for the input pixels of the input image based on the motion likelihood and the color likelihood, the segmentation indicators indicating whether individual input pixels are foreground input pixels or background input pixels.

2. The computer readable storage media of claim 1, the motion model comprising:

a plurality of spatial gradient magnitudes associated with the training pixels from the set of training images; and

a plurality of temporal derivatives associated with the training pixels from the set of training images.

3. The computer readable storage media of claim 1, wherein the motion likelihood is based on a likelihood of motion function:

U^M (α

,k^M,θ

^M,g,ż

).

4. The computer readable storage media of claim 1, the motion model comprising:

a Gaussian mixture model fitted to the distribution of the temporal derivatives and the spatial gradients of the training pixels from the set of training images.

5. The computer readable storage media of claim 1, wherein:

the color likelihood is based on at least one color model selected from the group consisting of a global color model and a pixel-wise color model, and

at least one of the segmentation indicators is selected from the group comprising opacity indicator values and fractional segmentation indicator values.

6. The computer readable storage media of claim 5, wherein the color likelihood is based on the equation U^C(α

,k,θ

,z)=Σ

D(α

_n,k_n,θ

,z_n).

7. A method comprising:

determining, on at least one computing device comprising at least one processing unit, a motion likelihood for input pixels of an input image, the motion likelihood being determined using a motion model without computing full velocities of the input pixels, the motion model having at least one parameter that is fitted to training images having training segmentation indicators distinguishing background pixels from foreground pixels in the training images;

determining, on the at least one computing device, a color likelihood for segmenting the input pixels of the input image based on a color likelihood model;

automatically generating, on the at least one computing device, a segmentation indicator associated with at least one of the input pixels, the segmentation indicator being based on the motion likelihood and the color likelihood; and

storing, on a computer storage media, the segmentation indicator.

8. The method of claim 7, further comprising:

determining, on the at least one computing device, a contrast likelihood based on the input pixels of the input image,

wherein the automatically generating the segmentation indicator is based on the contrast likelihood.

9. The method of claim 7, the training images comprising a series of images having a moving foreground and a static background.

10. The method of claim 7, further comprising:

fitting, on the at least one computing device, the at least one parameter of the motion model to a Gaussian mixture of a foreground distribution of a foreground temporal derivative and a foreground spatial gradient of the individual pixels in the training images; and

fitting, on the at least one computing device, the at least one parameter of the motion model to a Gaussian mixture of a background distribution of a background temporal derivative and a background spatial gradient of the individual pixels in the training images.

11. The method of claim 7, wherein the automatically generating the segmentation indicator includes:

optimizing the segmentation indicator based on a minimization of an energy function, the energy function based on the motion likelihood and the color likelihood.

12. The method of claim 11, wherein the optimizing the segmentation indicator includes using a graph cut.

13. The method of claim 11, wherein the energy function is a sum of the motion likelihood, the color likelihood, and a spatial smoothness likelihood.

14. The method of claim 7, further comprising:

modifying, on the at least one computing device, at least a portion of the input image based on the segmentation indicator.

15. The method of claim 14, wherein the modifying includes:

replacing at least one background pixel of the input image, the at least one background pixel having a segmentation indicator denoting background.

16. A system comprising:

a segmentation module configured to;

determine a motion likelihood for input pixels of an input image, the motion likelihood being determined using a motion model without explicitly calculating velocities of each of the input pixels, the motion model having at least one parameter that is fitted to training images having training segmentation indicators distinguishing background pixels from foreground pixels in the training images;

determine a color likelihood for segmenting the input pixels of the input image based on a color likelihood model; and

generate segmentation indicators associated with the input pixels, the segmentation indicators being generated based on a combination of the motion likelihood and the color likelihood; and

at least one processing unit configured to execute the segmentation module.

17. The system according to claim 16, the segmentation module being further configured to compute a sum of the motion likelihood and the color likelihood.

18. The system according to claim 16, the input image comprising monocular image data from a single monocular input device.

19. The system according to 16, the segmentation module being configured to generate the segmentation indicators by minimizing an energy function of at least the motion likelihood and the color likelihood.

20. The system according to claim 16, further comprising a motion initialization module configured to fit the at least one parameter of the motion model to the training images by minimizing classification error of the training segmentation indicators.