Segmentation of objects by minimizing global-local variational energy

• US 7,706,610 B2
• Filed: 11/29/2005
• Issued: 04/27/2010
• Est. Priority Date: 11/29/2005
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hua, Gang, Zhang, Zhengyou, Liu, Zicheng
• Primary Examiner(s): Ahmed; Samir A
• Assistant Examiner(s): Liu; Li
• Application Number:
  US11/291,309
• Publication Number:
  US 20070122039A1
• Time in Patent Office: 1,610 Days
• Field of Search: 382/173, 382/199
• US Class Current: 382/173
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06T 2207/20116 Active contour; Active surface; Snakes
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/149 involving deformable models, e.g. active contour models
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions

Abstract

An “Image Segmenter” provides a variational energy formulation for segmentation of natural objects from images. In general, the Image Segmenter operates by adopting Gaussian mixture models (GMM) to capture the appearance variation of objects in one or more images. A global image data likelihood potential is then computed and combined with local region potentials to obtain a robust and accurate estimation of pixel foreground and background distributions. Iterative minimization of a “global-local energy function” is then accomplished by evolution of a foreground/background boundary curve by level set, and estimation of a foreground/background model by fixed-point iteration, termed “quasi-semi-supervised EM.” In various embodiments, this process is further improved by providing general object shape information for use in rectifying objects segmented from the image.

Claims

1. A system for automatically identifying a boundary curve for delimiting an object of interest within an image frame, comprising using a computing device to perform the steps for:

receiving an image frame containing an object of interest;

sampling separate areas of the image frame to initialize separate probabilistic color distribution models of a single foreground region and a single background region of the image frame, said probabilistic models of the foreground and background regions jointly comprising a local image data likelihood covering the entire image;

initializing a global image data likelihood as a function of a combination of the probabilistic models of the foreground and background regions of the image frame;

initializing a boundary curve as a boundary surrounding the area of the image frame sampled to initialize the probabilistic model of the foreground region; and

jointly performing an iterative minimization of energy functionals representing the boundary curve, the local image data likelihood and the global image data likelihood to generate a final boundary curve for delimiting the object of interest.

2. The system of claim 1 further comprising refining the final boundary curve as a function of prior knowledge of object shapes.

3. The system of claim 1 further comprising segmenting the object of interest from the image frame with respect to the final boundary curve.

4. The system of claim 1 wherein sampling the image frame to initialize the probabilistic model of the background region comprises sampling image pixels in a plurality of regions along an outer boundary of the image frame.

5. The system of claim 1 wherein sampling the image frame to initialize the probabilistic model of the foreground region comprises sampling a plurality of image pixels in a region located in the center of the image frame.

6. The system of claim 1 wherein sampling the image frame to initialize the probabilistic model of the foreground region comprises sampling a plurality of image pixels in a region of the image selected via a user interface, said user selected region at least partially overlapping the object of interest.

7. The system of claim 1 wherein the iterative minimization of the energy functionals comprises repeating steps for:

minimizing an energy of the boundary curve with the local and global probabilistic models fixed; and

minimizing an energy of the local and global probabilistic models with the boundary curve fixed.

8. The system of claim 7 wherein the iterative steps for minimization of the energy functionals repeats until any of:

a convergence is achieved; and

a maximum number of iterations have been completed.

9. A physical computer readable medium having computer executable instructions stored therein for automatically segmenting an object of interest from an image frame, said computer executable instructions comprising:

sampling an input image in a plurality of regions along an outer edge of the input image to construct an initial single Gaussian mixture model representing a color distribution of the input image background;

sampling the input image in a region assumed to include a least a portion of the object of interest to construct an initial single Gaussian mixture model representing a color distribution of the input image foreground;

initializing a boundary curve as a curve encompassing the region sampled for construction of the foreground Gaussian mixture model;

initializing a single global Gaussian mixture model as a weighted function of the foreground and background Gaussian mixture models covering the entire input image;

constructing a local energy functional from a weighted combination of the foreground and background Gaussian mixture models;

constructing a global energy functional from the global Gaussian mixture model;

jointly performing an iterative minimization of an energy functional representing the boundary curve, the local energy functional, and the global energy functional for evolving a final boundary curve for delimiting the object of interest; and

segmenting the object of interest from the input image as a function of the final boundary curve.

10. The computer-readable medium of claim 9 wherein sampling the input image in a region assumed to include a least a portion of the object of interest comprises automatically sampling the image pixels in a region located at the center of the input image.

11. The computer-readable medium of claim 9 wherein sampling the input image in a region assumed to include a least a portion of the object of interest comprises sampling a region of the input image identified via a user interface.

12. The computer-readable medium of claim 9 wherein jointly performing an iterative minimization of energy functionals representing the boundary curve, the local energy functional, and the global energy functional computer executable instructions steps for:

minimizing an energy of the boundary curve with the local and global energy functionals held fixed; and

minimizing an energy of the local and global energy functionals with the boundary curve fixed.

13. The computer-readable medium of claim 9 wherein the joint iterative minimization of the energy functionals continues until any of:

achieving a convergence of energy functional values; and

completing a predefined maximum number of iterations.

14. A computer implemented process for identifying an object of interest within an image, comprising using a computer to:

sample an image to construct an initial single Gaussian mixture model representing a color distribution of the image background;

sample a region of the image to construct an initial single Gaussian mixture model representing a color distribution of the image foreground;

initialize a boundary curve to encompass the region sampled for construction of the foreground Gaussian mixture model;

initialize a global Gaussian mixture model as a weighted function of the foreground and background Gaussian mixture models covering the entire image;

jointly performing an iterative minimization of the energy of;

the boundary curve,

a combination of the foreground and background Gaussian mixture models, said combination representing a local Gaussian mixture model, and

the global Gaussian mixture model; and

wherein the joint iterative minimization results in the evolution of a final boundary curve for delimiting the object of interest.

15. The computer implemented process of claim 14 further comprising segmenting the object of interest from the image as a function of the final boundary curve.

16. The computer implemented process of claim 14 wherein performing the joint iterative minimization comprises repeating a set of steps including:

minimizing an energy of the boundary curve with the local and global Gaussian mixture models held fixed; and

minimizing an energy of the local and global Gaussian mixture models with the boundary curve held fixed.

17. The computer implemented process of claim 16 wherein the set of steps is repeated until such time as convergence of the local and global Gaussian mixture models and has been reached.

18. The computer implemented process of claim 16 wherein the set of steps is repeated until such time as a predefined maximum number of iterations has been reached.

19. The computer implemented process of claim 14 wherein sampling the image to construct the initial Gaussian mixture model of the image background comprises sampling image pixels in a plurality of regions along an outer boundary of the image.

20. The computer implemented process of claim 14 wherein sampling the region of the image to construct the initial Gaussian mixture model of the image foreground comprises sampling any of:

the image pixels in a region located in the center of the image; and

the image pixels in a user specified region of the image.