IMAGE SEGMENTATION

• US 20100119147A1
• Filed: 01/19/2010
• Published: 05/13/2010
• Est. Priority Date: 06/17/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cross, Geoffrey, Criminisi, Antonio, Kolmogorov, Vladimir, Blake, Andrew
• Granted Patent: US 8,103,093 B2
• Time in Patent Office: Days
• US Class Current: 382/164
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20121 Active appearance model [AAM]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

Segmentation of foreground from background layers in an image may be provided by a segmentation process which may be based on one or more factors including motion, color, contrast, and the like. Color, motion, and optionally contrast information may be probabilistically fused to infer foreground and/or background layers accurately and efficiently. A likelihood of motion vs. non-motion may be automatically learned from training data and then fused with a contrast-sensitive color model. Segmentation may then be solved efficiently by an optimization algorithm such as a graph cut. Motion events in image sequences may be detected without explicit velocity computation.

Claims

1. A computer readable storage media containing computer readable instructions that, when executed by a computing device, cause the computing device to perform steps, the steps comprising:

determining a motion likelihood for each of a plurality of pixels of an image, the motion likelihood determined without a velocity of the pixels;

determining a color likelihood of the plurality of pixels of the image; and

automatically generating a segmentation indicator for each of the plurality of input pixels of the image based on the motion likelihood and the color likelihood.

2. The computer readable storage media of claim 1, wherein the motion likelihood is based on a motion model, the motion model comprising:

at least one motion parameter;

a plurality of spatial gradient magnitudes, each of the plurality of spatial gradient magnitudes associated with at least one of the segmentation indicators, each of the plurality of spatial gradient magnitudes associated with at least one of the plurality of pixels; and

a plurality of temporal derivatives, each of the plurality of spatial gradient magnitudes associated with at least one of the segmentation indicators, each of the plurality of spatial gradient magnitudes associated with at least one of the plurality of pixels.

3. The computer readable storage media of claim 1, wherein the motion likelihood is based on a likelihood of motion function:

U^M(α

,k^M,θ

^M,g,ż

).

4. The computer readable storage media of claim 1, the motion likelihood based on a likelihood of motion function, the likelihood of motion function comprising:

fitting a Gaussian mixture model to a distribution of a temporal derivative and a spatial gradient of a plurality of pixels in each image in a set of training images.

5. The computer readable storage media of claim 1, wherein:

the color likelihood is based on at least one color model selected from the group consisting of a global color model and a pixel-wise color model, and

at least one of the segmentation indicators is selected from the group comprising foreground segmentation indicator, background segmentation indicator, opacity indicator values, and fractional segmentation indicator values

6. The computer readable storage media of claim 5, wherein the color likelihood is based on the equation U^C(α

,k,θ

,z)=Σ

D(α

_n,k_n,θ

,z_n).

7. A method comprising:

determining, on at least one computing device comprising at least one processing unit, a motion likelihood for each of a plurality of pixels of an image, the motion likelihood determined without full velocities of the plurality of pixels;

determining, on the at least one computing device, a color likelihood for each of the plurality of pixels of the image based on a color likelihood model;

automatically generating, on the at least one computing device, a segmentation indicator associated with at least one of the plurality of pixels, the segmentation indicator based on the motion likelihood and the color likelihood; and

storing, on a computer storage media, the segmentation indicator.

8. The method of claim 7, further comprising:

determining, on the at least one computing device, a contrast likelihood based on the plurality of pixels in the image,

wherein the automatically generating the segmentation indicator is based on the contrast likelihood.

9. The method of claim 7, wherein the motion likelihood is based on a motion likelihood parameter, the motion likelihood parameter is based on training data comprising a series of images, the image different than the series of images.

10. The method of claim 7, wherein the motion likelihood is based on at least one motion likelihood parameter, the at least one motion likelihood parameter based on:

fitting, on the at least one computing device, a Gaussian mixture model to a foreground distribution of a foreground temporal derivative and a foreground spatial gradient of a plurality of pixels in each image in a set of training images; and

fitting, on the at least one computing device, a Gaussian mixture model to a background distribution of a background temporal derivative and a background spatial gradient of the plurality of pixels in each image in the set of training images.

11. The method of claim 7, wherein the generating the segmentation indicator includes:

optimizing the segmentation indicator for each of the plurality of input pixels based on a minimization of an energy function, the energy function based on the motion likelihood and the color likelihood.

12. The method of claim 11, wherein the optimizing the segmentation indicator includes using a graph cut.

13. The method of claim 11, wherein the energy function is a sum of the motion likelihood, the color likelihood, and a spatial smoothness likelihood.

14. The method of claim 7, further comprising:

modifying, on the at least one computing device, at least a portion of the image based on the segmentation indicator.

15. The method of claim 14, wherein the modifying includes:

replacing at least one pixel of the image, the at least one pixel having a segmentation indicator denoting background.

16. A system comprising:

means for determining a spatial smoothness of an input image;

means for determining a motion likelihood of each of a plurality of pixels of the input image, the motion likelihood determined without determining a velocity for the pixels;

means for determining a color likelihood of each of the plurality of pixels; and

means for generating at least one label associated with at least one pixel of the plurality of pixels, the at least one label indicating a segmentation indicator for the at least one pixel.

17. The system of claim 16, wherein the means for generating include:

a means for probabilistically fusing the means for determining spatial smoothness, the means for determining a motion likelihood, and the means for determining a color likelihood.

18. The system of claim 16, further comprising:

means for training at least one motion likelihood parameter based on a first training image different from the input image.

19. The system of claim 18, wherein a difference between the first training image and the input image is selected from the group comprising a location, a number of people, and a background color.

20. The system of claim 16, further comprising:

means for modifying the input image based on the generated labels.