BACKGROUND MAINTENANCE OF AN IMAGE SEQUENCE

• US 20060193516A1
• Filed: 05/10/2006
• Published: 08/31/2006
• Est. Priority Date: 09/20/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Toyama, Kentaro, Krumm, John C., Meyers, Brian R., Brumitt, Barry L.
• Granted Patent: US 7,577,294 B2
• Time in Patent Office: Days
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

A system and method for maintaining a background model of an image sequence by processing on multiple spatial scales. These multiple spatial scales include a pixel scale, a regional scale and a frame scale. The image sequence undergoes pixel processing that determines a current background model and provides an initial pixel assignment as either a background or a foreground pixel. Region processing further refines the initial pixel assignments by considering relationships between pixels and possibly reassigning pixels. Frame processing further refines the current background model by determining whether a substantial change has occurred in the actual background and, if so, providing a more accurate background model.

Claims

1. A background maintenance system for maintaining a background model of an image sequence having a plurality of pixels, comprising:

a prediction module that provides at least two predictions of what value a pixel will be in a next frame of the image sequence;

a first pixel value prediction that predicts what value the pixel will be in the next frame of the image sequence based on actual history pixel values; and

a second pixel value prediction that predicts what value the pixel will be in the next frame of the image sequence based on predicted history pixel values.

2. The background maintenance system of claim 1, further comprising:

a computation module that determines an actual value of the pixel in the next frame of the image sequence; and

a declaration module that classifies the pixel as a foreground pixel if the actual value of the pixel in the next frame of the image sequence differs from any one of the first and second pixel value predictions by more than a threshold value.

3. The background maintenance system of claim 1, further comprising at least one refinement module that processes the image sequence on a spatial scale other than the pixel scale.

4. The background maintenance system of claim 3, wherein the refinement module includes a first refinement module that is a region processing module that processes the image sequence on a regional scale.

5. The background maintenance system of claim 4, wherein the region processing module further comprises considering a relationship between at least some of the plurality of pixels to provide pixel assignment.

6. The background maintenance system of claim 4, wherein the refinement module includes a second refinement module that is a frame processing module that processes the image sequence on a frame scale.

7. The background maintenance system of claim 6, wherein the frame processing module further determines a background model that most accurately represents an actual background of the image sequence and performs one of:

(a) retaining a current background model;

(b) substituting a more accurate background model in place of the current background model.

8. The background maintenance system of claim 1, further comprising an update module that stores and appends the actual history pixel values in a list of actual history pixel values.

9. The background maintenance system of claim 1, further comprising an update module that stores and appends the predicted history pixel values in list of predicted history pixel values

10. A computer-readable medium having computer-executable modules, comprising:

a pixel processing module that processes an image sequence on a pixel scale and further comprising;

a prediction module that provides two history-based predictions of what value a particular pixel will be in a next frame of the image sequence to be processed, the two history-based predictions further comprising;

a first history-based pixel value prediction that predicts a pixel value of the particular pixel in the next frame of the image sequence based on actual history pixel values;

a second history-based pixel value prediction that predicts a pixel value of the particular pixel in the next frame of the image sequence based on predicted history pixel values; and

at least one refinement module that processes the image sequence on a spatial scale other than the pixel scale.

11. The computer-readable medium of claim 10, wherein the refinement module processes the image sequence on scale larger than the pixel scale.

12. The computer-readable medium of claim 11, wherein a first refinement module is a region processing module that processes the image sequence on a region scale.

13. The computer-readable medium of claim 12, wherein a second refinement module is a frame processing module that processes the image sequence on a frame scale.

14. A method for maintaining a background model of an image sequence having a plurality of pixels, comprising:

predicting what value a pixel will have in a next frame of the image sequence using a list of actual history pixel values to generate a first pixel value prediction;

predicting what value the pixel will have in the next frame of the image sequence using a list of predicted history pixel values to generate a second pixel value prediction; and

processing the image sequence on a pixel scale using the first pixel value prediction and the second pixel value prediction to determine a current background model and provide an initial assignment for the pixel.

15. The method of claim 14, further comprising refining the pixel processing by processing on a spatial scale other that the pixel scale to further refine at least one of:

(a) the current background model;

(b) the initial pixel assignment.

16. The method of claim 15, wherein refining further comprises providing a region processing module that processes the image sequence on a region scale.

17. The method of claim 16, wherein refining further comprises providing a frame processing module that processes the image sequence on a frame scale.

18. The method of claim 17, wherein refining further comprises providing a postprocessing module that enhances the image sequence.

19. The method of claim 18, wherein the postprocessing module enhances the image sequence by providing speckle removal.

20. The method of claim 14, further comprising:

determining an actual value of the pixel in the next frame of the image sequence; and

classifying the pixel as a foreground pixel if the actual value of the pixel in the next frame of the image sequence differs from any one of the first and second pixel value predictions by more than a threshold value.