GENERATIVE MODELS FOR CONSTRUCTING PANORAMAS FROM AN IMAGE SEQUENCE

• US 20100238266A1
• Filed: 06/06/2010
• Published: 09/23/2010
• Est. Priority Date: 11/14/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Frey, Brendan J., Jojic, Nebojsa
• Granted Patent: US 7,940,264 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

A simplified general model and an associated estimation algorithm is provided for modeling visual data such as a video sequence. Specifically, images or frames in a video sequence are represented as collections of flat moving objects that change their appearance and shape over time, and can occlude each other over time. A statistical generative model is defined for generating such visual data where parameters such as appearance bit maps and noise, shape bit-maps and variability in shape, etc., are known. Further, when unknown, these parameters are estimated from visual data without prior pre-processing by using a maximization algorithm. By parameter estimation and inference in the model, visual data is segmented into components which facilitates sophisticated applications in video or image editing, such as, for example, object removal or insertion, tracking and visual surveillance, video browsing, photo organization, video compositing, etc.

Claims

1. A method for automatically creating a panoramic view of a scene, comprising using a computing device to perform steps for:

receiving a video sequence of a scene captured from a moving camera;

zero padding borders of each image frame of the video sequence to increase extents of each image frame;

setting a preferred number sprites to be identified within the zero padded video sequence;

setting a preferred number of layers into which each frame of the zero padded video sequence is to be decomposed, said layers including at least a background layer and a front layer, said front layer passing the entire contents of each zero padded image frame to be probabilistically considered for contributions to the background layer via a generative model;

using the generative model to automatically decompose the zero padded video sequence into the preferred number of sprites and the preferred number of layers, said layers including the background layer; and

wherein the background layer decomposed from the zero padded video sequence provides a static panoramic view of the scene relative to the original image frames of the video sequence.

2. The method of claim 1 further comprising constructing a composite panoramic image by compositing one or more of the decomposed sprites into the background layer.

3. The method of claim 1 wherein one or more of the decomposed sprites are “

flexible sprites”

, wherein each flexible sprite represents a corresponding moving object having a geometry that is dynamically changeable from frame to frame of the zero padded video sequence, such that each flexible sprite provides a masked sequence of images of a corresponding moving object extracted from the zero padded video sequence.

4. The method of claim 3 further comprising constructing a composite output image sequence by compositing one or more of the flexible sprites into a new sequence of image frames constructed by repeating the static panoramic view of the scene.

5. The method of claim 1 wherein setting the preferred number sprites comprises automatically determining an optimal number of sprites from the zero padded video sequence.

6. The method of claim 1 wherein setting the preferred number of layers comprises automatically determining an optimal number of layers from the zero padded video sequence.

7. The method of claim 1 wherein automatically decomposing the zero padded video sequence into the preferred number of sprites and the preferred number of layers comprises performing a probabilistic variational expectation-maximization analysis of the zero padded video sequence.

8. The method of claim 1 wherein the generative model is automatically constructed to represent the entire zero padded video sequence, and wherein the generative model is formed using a set of automatically determined model parameters comprising:

a prior probability of at least one sprite class;

means and variances of sprite appearance maps; and

means and variances of sprite masks.

9. A computer-implemented process for constructing a panoramic view of a scene background from an image sequence captured from a moving camera, comprising using a computing device to:

receive an image sequence of a scene captured from a moving camera;

zero pad borders of each image frame of the image sequence to increase extents of each image frame, and wherein increasing the amount of zero padding will increase extents of a background layer;

set a preferred number of layers into which each frame of the image sequence is to be decomposed, said layers including at least the background layer and a front layer, said front layer passing the entire contents of each zero padded image frame to be probabilistically considered for contributions to the background layer via a generative model; and

use the generative model to automatically decompose the zero padded image sequence into the preferred number of layers, said layers including the background layer, said background layer providing a static panoramic view of the scene relative to the frames of the original image sequence.

10. The computer-implemented process of claim 9 further comprising setting a preferred number of zero or more objects to be identified within the image sequence, and wherein the generative model further decomposes the zero padded image sequence into the preferred number of objects.

11. The computer-implemented process of claim 10 further comprising constructing a composite panoramic image by compositing one or more of the decomposed objects into the background layer.

12. The computer-implemented process of claim 10 wherein one or more of the decomposed objects are “

flexible sprites”

, wherein each flexible sprite represents a corresponding moving object having a geometry that is dynamically changeable from frame to frame of the zero padded image sequence, such that each flexible sprite provides a masked sequence of images of a corresponding moving object extracted from the zero padded image sequence.

13. The computer-implemented process of claim 12 further comprising constructing a composite output image sequence by compositing one or more of the flexible sprites into a new sequence of image frames constructed by repeating the static panoramic view of the scene represented by the background layer.

14. The computer-implemented process of claim 9 wherein setting the preferred number of layers comprises automatically determining an optimal number of layers from the zero padded image sequence.

15. The computer-implemented process of claim 9 wherein automatically decomposing the zero padded video sequence into the preferred number of layers comprises using the generative model for performing a probabilistic variational expectation-maximization analysis of the zero padded image sequence.

16. A computer-readable storage device having computer executable instructions stored thereon for automatically constructing a panoramic view of a scene background from an image sequence, comprising computer executable instructions for:

receiving a video sequence of a scene captured from a moving camera;

zero padding borders of each image frame of the video sequence to increase extents of each image frame;

setting a preferred number sprites to be identified within the zero padded video sequence;

setting a preferred number of layers into which each frame of the zero padded video sequence is to be decomposed, said layers including at least a background layer and a front layer, said front layer passing the entire contents of each zero padded image frame to be probabilistically considered for contributions to the background layer via a probabilistic generative model;

using the generative model to automatically decompose the zero padded video sequence into the preferred number of sprites and the preferred number of layers, said layers including the background layer; and

wherein the background layer decomposed from the zero padded video sequence provides a static panoramic view of the scene relative to the original image frames of the video sequence.

17. The computer-readable storage device of claim 16 further comprising instructions for constructing a composite panoramic image by compositing one or more of the decomposed sprites into the background layer.

18. The computer-readable storage device of claim 16 wherein one or more of the decomposed sprites are “

flexible sprites”

, wherein each flexible sprite represents a corresponding moving object having a geometry that is dynamically changeable from frame to frame of the zero padded video sequence, such that each flexible sprite provides a masked sequence of images of a corresponding moving object extracted from the zero padded video sequence.

19. The computer-readable storage device of claim 18 further comprising instructions for constructing a composite output image sequence by compositing one or more of the flexible sprites into a new sequence of image frames constructed by repeating the static panoramic view of the scene represented by the background layer.

20. The computer-readable storage device of claim 16 wherein automatically decomposing the zero padded video sequence into the preferred number of sprites and the preferred number of layers is accomplished by performing a probabilistic variational expectation-maximization analysis of the zero padded video sequence to determine optimal numbers of layers and sprites.