Image segmentation using spatial-color gaussian mixture models

• US 20070237393A1
• Filed: 03/30/2006
• Published: 10/11/2007
• Est. Priority Date: 03/30/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Zhang, Cha, Cohen, Michael, Yu, Ting
• Granted Patent: US 7,885,463 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

A spatial-color Gaussian mixture model (SCGMM) image segmentation technique for segmenting images. The SCGMM image segmentation technique specifies foreground objects in the first frame of an image sequence, either manually or automatically. From the initial segmentation, the SCGMM segmentation system learns two spatial-color Gaussian mixture models (SCGMM) for the foreground and background objects. These models are built into a first-order Markov random field (MRF) energy function. The minimization of the energy function leads to a binary segmentation of the images in the image sequence, which can be solved efficiently using a conventional graph cut procedure

Claims

1. A process for segmenting an image of an image sequence, comprising the process actions of:

(a) designating foreground regions in an initial image of an image sequence;

(b) defining certain regions of the initial image as foreground and certain regions of the initial image as background based on the designated foreground regions;

(c) learning a foreground spatial-color Gaussian Mixture Model (SCGMM) model which describes the foreground regions and background SCGMM model that describes the background regions of the initial image;

(d) segmenting the initial image using a graph cut procedure which minimizes a Markov random field energy function containing the learned foreground and background SCGMM models;

(e) inputting a next image of the image sequence;

(f) updating the foreground and background SCGMM models using a joint SCGMM tracking procedure; and

(g) performing a graph cut procedure which minimizes a Markov Random Field energy function containing the updated foreground and background SCGMM models to segment the next image of the sequence; and

(h) updating each of the foreground and background SCGMM models with the segmented result of process action (g).

2. The process of claim 1 further comprising repeating process actions (e) through (h) to segment subsequent images in the image sequence.

3. The process of claim 1 wherein the process action of updating the foreground and background SCGMM models using a joint tracking procedure comprises:

combining the foreground and background SCGMM models of a previous image into a generative model;

running a constrained Expectation Maximization procedure to maximize the joint likelihood of the next input image; and

separating the generative model back into an updated foreground SCGMM model and an updated background SCGMM model.

4. The process of claim 3 wherein the process action of running a constrained Expectation Maximization procedure to maximize the joint likelihood comprises:

fixing color components of the generative model; and

updating spatial components and weights of the spatial components of the generative model.

5. The process of claim 4 wherein the weights of the spatial components are determined by the percentage of the foreground and background regions.

6. The process of claim 1 wherein each said energy function minimized in a graph cut procedure comprises a two-way graph cut algorithm having two terminal nodes that represent the foreground and background SCGMM models.

7. The process of claim 1 wherein the designating process action comprises manually designating the foreground regions.

8. The process of claim 1 wherein the designating process action comprises automatically designating the foreground regions.

10. The process of claim 1 wherein the process actions (f) through (h) are repeated more than once for an input image.

11. A computer-readable medium having computer-executable instructions for performing the process recited in claim 1.

9. The process of claim 9 wherein the foreground regions correspond to a face and wherein the face is automatically designated by a face detector.

12. A system for segmenting a sequence of images into foreground and background regions, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

segment an initial image of an image sequence by identifying foreground objects in the image and using a first graph cut procedure which minimizes a Markov random field energy function containing foreground and background spatial color Gaussian Mixture Model (SCGMM) models;

update the foreground and background SCGMM models for a next image in the image sequence using a joint SCGMM tracking procedure; and

perform a second graph cut procedure which minimizes a Markov Random Field energy function containing the updated foreground and background SCGMM models to segment the next image of the sequence.

13. The system of claim 12 further comprising a module for updating each of the foreground and background SCGMM models with the segmented result of the next image of the sequence.

14. The system of claim 12 wherein the module to segment the initial image comprises sub-modules to:

designate foreground objects in said initial image;

define certain objects of the initial image as foreground and certain objects of the initial image as background;

learn a foreground and a background spatial-color Gaussian Mixture Model (SCGMM) model which describes the foreground objects and background objects of the initial image; and

segment the initial image using a graph cut procedure which minimizes a Markov random field energy function containing the learned foreground and background SCGMM models.

15. The system of claim 12 wherein the module to update the foreground and background SCGMM models using a joint tracking procedure comprises sub-modules to:

combine the foreground and background SCGMM models of a previous image with corresponding weights relative to the coverage size of the foreground and background objects of the previous image to create a generative model; and

run a constrained Expectation Maximization procedure to maximize the joint likelihood of the next image of the image sequence using the generative model.

16. The system of claim 15 wherein the sub-module to run a constrained Expectation Maximization procedure to maximize the joint likelihood comprises sub-modules to:

fix the color factors of the foreground and background SCGMM models combined into the generative model; and

update the mean and variance portion of the spatial portion of each foreground and background SGCMM models combined into the generative model.

17. A process for segmenting an image of an image sequence, comprising the process actions of:

inputting an initial video frame of a video sequence;

applying an object detector to the video frame to determine foreground objects in the video frame;

defining objects of the video frame as foreground and objects of the video frame as background based on the objects detected by the object detector;

learning a foreground spatial-color Gaussian Mixture Model (SCGMM) model which describes the foreground objects of the video frame and a background SCGMM model which describes the background objects of the video frame; and

segmenting the video frame using a graph cut procedure which minimizes a Markov random field energy function containing the learned foreground and background SCGMM models.

18. The process of claim 17 wherein the object detector is a face detector and wherein the detected object is a face.

19. The process of claim 17 wherein the Markov random field energy function is minimized by a graph cut algorithm with two terminal nodes that represent the foreground and the background.

20. A computer-readable medium having computer-executable instructions for performing the process recited in claim 17.