STABILIZATION OF OBJECTS WITHIN A VIDEO SEQUENCE

• US 20070104383A1
• Filed: 09/23/2006
• Published: 05/10/2007
• Est. Priority Date: 11/14/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Jojic, Nebojsa, Frey, Brendan
• Granted Patent: US 7,680,353 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

A simplified general model and an associated estimation algorithm is provided for modeling visual data such as a video sequence. Specifically, images or frames in a video sequence are represented as collections of flat moving objects that change their appearance and shape over time, and can occlude each other over time. A statistical generative model is defined for generating such visual data where parameters such as appearance bit maps and noise, shape bit-maps and variability in shape, etc., are known. Further, when unknown, these parameters are estimated from visual data without prior pre-processing by using a maximization algorithm. By parameter estimation and inference in the model, visual data is segmented into components which facilitates sophisticated applications in video or image editing, such as, for example, object removal or insertion, tracking and visual surveillance, video browsing, photo organization, video compositing, etc.

Claims

1. A system for stabilizing an image sequence, comprising:

receiving an input image sequence comprising two or more sequential image frames of a scene;

evaluating the frames of the input image sequence and automatically learning a probabilistic number of objects to be identified within each frame of the input image sequence and a probabilistic number of layers for each frame;

probabilistically decomposing the input image sequence into the learned number of objects and layers;

constructing a stabilized output image sequence by using one or more of the decomposed objects and layers from two or more frames of the input image sequence to create a composite sequence of stabilized image frames; and

wherein corresponding objects and corresponding layers are transformed to construct a sequential stabilized alignment of objects and layers between each frame of the stabilized output image sequence.

2. The system of claim 1 wherein the objects and layers of the decomposed input image sequence are user selectable, and wherein the stabilized image sequence is constructed using only user selected objects and layers.

3. The system of claim 1 wherein the objects and layers of the decomposed input image sequence are user selectable, and wherein the stabilized image sequence is constructed using one or more objects and layers that are not selected by the user.

4. The system of claim 1 wherein probabilistically decomposing the input image sequence into the learned number of objects and layers further includes probabilistically inferring transformations for each object and layer from one frame to the next.

5. The system of claim 4 wherein constructing the stabilized alignment of objects and layers between each frame includes applying an inverse transformation and alignment to each corresponding object and layer for stabilizing each object and layer within the composite sequence of stabilized image frames.

6. The system of claim 1 wherein probabilistically decomposing the input image sequence into the learned number of objects comprises automatically learning a 2-dimensional flexible representation of each object and an image mask for each 2-dimensional flexible representation of each object.

7. The system of claim 1 wherein probabilistically decomposing the input image sequence into the learned number of objects and layers comprises performing a probabilistic variational expectation-maximization analysis of the frames of the input image sequence, said variational expectation-maximization analysis further comprising:

learning a plurality of probabilistic submodels having variational parameters representing posterior distributions for each frame of the input image sequence;

learning at least one probabilistic model representing each frame of the input image sequence from a combination of the learned submodels, with two or more submodels sharing common interface variables in the model; and

iteratively evaluating the probabilistic model by iteratively evaluating the submodels using a variational generalized expectation-maximization analysis to optimize a lower bound on posterior distributions of the learned submodels.

8. A computer-implemented process for automatically stabilizing a layered representation of an image sequence, comprising using a computing device to:

acquire an image sequence having a plurality of image frames;

automatically decompose the image sequence into a generative model, said generative model including a set of model parameters that represent at least one probabilistic object and at least one image layer learned for each image frame;

automatically regenerating a stabilized layered representation of the image sequence by using the generative model to construct an output image sequence corresponding to one or more of the frames of the input image sequence; and

wherein constructing the output image sequence comprises using the model parameters for performing an inverse transformation and alignment of one or more of the learned probabilistic objects and image layers from one or more sequential frames of the input image sequence for stabilizing those objects and layers relative to new sequential image frames comprising the output image sequence.

9. The computer-implemented process of claim 8 wherein the objects and layers of the decomposed image sequence are user selectable.

10. The computer-implemented process of claim 9 wherein automatically constructing the output image sequence further comprises using only user selected objects and layers to construct the output image sequence.

11. The computer-implemented process of claim 9 wherein automatically constructing the output image sequence further comprises using only objects and layers not selected by the user to construct the output image sequence.

12. The computer-implemented process of claim 8 wherein automatically decomposing the image sequence into a generative model further comprises decomposing the image sequence relative to user specified numbers of objects and layers.

13. The computer-implemented process of claim 8 wherein the model parameters of each generative model include an object appearance map, an object mask, and a prior probability of at least one object class.

14. The computer-implemented process of claim 8 wherein automatically regenerating a stabilized layered representation of the image sequence further comprises automatically centering at least one object in one or more of the new sequential image frames comprising the output image sequence.

15. A computer-readable medium having computer executable instructions for automatically learning and stabilizing layered flexible image sprites in an input image sequence, comprising:

receiving an image sequence of at least two image frames of a scene;

automatically learning a probabilistic number of image sprite classes to be identified within the image sequence;

automatically learning a probabilistic dimensionality of each image sprite class;

automatically learning a probabilistic number of layers for the image sprite classes;

automatically learning at least one layered image sprite from the image sequence given the automatically learned number of image sprite classes, image sprite dimensionality, and image layers, wherein each image sprite represents an object in the image sequence;

automatically constructing a generative model from the automatically learned layered image sprites, wherein the generative model includes a set of model parameters that represent the input sequence; and

using the generative model for reconstructing at least one sequential image frame from at least one of the objects in the image sequence by performing an inverse transformation and alignment of corresponding objects from one or more sequential frames of the image sequence to create an output sequence.

16. The computer-readable medium of claim 15 wherein learned objects are user selectable.

17. The computer-readable medium of claim 16 wherein using the generative model for reconstructing at least one sequential image frame further comprises using only user selected objects.

18. The computer-readable medium of claim 16 wherein using the generative model for reconstructing at least one sequential image frame further comprises using only objects not selected by the user.

19. The computer-readable medium of claim 15 wherein automatically constructing the generative model further comprises performing a probabilistic variational expectation-maximization analysis of the frames of the input image sequence, said variational expectation-maximization analysis further comprising:

learning a plurality of probabilistic submodels having variational parameters representing posterior distributions for each frame of the input image sequence;

learning at least one probabilistic model representing each frame of the input image sequence from a combination of the learned submodels, with two or more submodels sharing common interface variables in the learned probabilistic model; and

optimizing the learned probabilistic model by iteratively evaluating the submodels using a variational generalized expectation-maximization analysis to optimize a lower bound on posterior distributions of the learned submodels.

20. The computer-readable medium of claim 15 wherein the model parameters include:

a prior probability of at least one sprite class;

means and variances of sprite appearance maps;

means and variances of sprite masks; and

observation noise variances.