Background maintenance of an image sequence

• US 20060045335A1
• Filed: 10/22/2005
• Published: 03/02/2006
• Est. Priority Date: 09/20/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Toyama, Kentaro, Krumm, John C., Meyers, Brian R., Brumitt, Barry L.
• Granted Patent: US 7,548,648 B2
• Time in Patent Office: Days
• US Class Current: 382/171
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

An image background maintenance system and method for maintaining a background model of an image sequence by processing on multiple spatial scales. These multiple spatial scales include a pixel scale, a regional scale and a frame scale. The image sequence undergoes pixel processing that determines a current background model and provides an initial pixel assignment as either a background or a foreground pixel. Region processing further refines the initial pixel assignments by considering relationships between pixels and possibly reassigning pixels. Frame processing further refines the current background model by determining whether a substantial change has occurred in the actual background and, if so, providing a more accurate background model.

Claims

1. An image processing system for processing an image having pixels, comprising:

an intersection module that groups the pixels to form an intersection image based on an initial classification of each pixel;

a histogram module that determines a property of the pixels within the intersection image to create a boundary of a foreground region; and

a backprojection module that fills in the foreground region within the boundary to segment the foreground region from the image.

2. The image processing system of claim 1, further comprising an image differencing module that identifies moving pixels using a plurality of previous images earlier in time than the image.

3. The image processing system of claim 2, wherein:

the image differencing module takes the difference of the plurality of previous images that are adjacent in time to form a differenced image; and

the intersection module uses the differenced image to form the intersection image that contains moving foreground pixels.

4. The image processing system of claim 3, wherein the intersection module forms the intersection image by calculating the intersection of the differenced image and a foreground image containing pixels classified as foreground pixels.

5. The image processing system of claim 3, wherein the histogram module calculates the property of the pixels in the intersection image.

6. The image processing system of claim 5, wherein the property of the pixels in the intersection image are calculated using pixel property values from a previously processed frame.

7. The image processing system of claim 6, wherein the property of the pixels is at least one of:

(a) pixel color;

(b) pixel intensity;

(c) pixel depth.

8. The image processing system of claim 6, wherein a frequency threshold is defined such that pixels occurring less than the frequency threshold are not included in the foreground region.

9. The image processing system of claim 5, wherein the backprojection module uses connectivity to determine which pixels to include within the boundary.

10. A computer-readable medium having computer-executable modules, comprising:

an intersection module that groups pixels in a current frame based on an initial classification of each pixel to form an intersection image;

a histogram module that determines a property of intersection image pixels to form a boundary of a foreground region; and

a backprojection module that connects pixels within the boundary of the foreground region to divide the foreground region from the current frame.

11. The apparatus of claim 10, further comprising an image differencing module that identifies moving pixels using a plurality of frames, each of the plurality of frames being earlier in time that the current frame.

12. The apparatus of claim 11, wherein:

the image differencing module calculates the difference between temporally adjacent frame to form a differenced image; and

the intersection module uses the differenced image to group the pixels.

13. The apparatus of claim 12, wherein the intersection module calculates the intersection of the differenced image and a foreground image containing foreground pixels to form the intersection image containing moving foreground pixels.

14. The apparatus of claim 11, wherein the histogram module calculates a histogram for a property of each pixel within the intersection image.

15. The apparatus of claim 14, wherein the pixel property is at least one of:

(a) pixel color;

(b) pixel intensity;

(c) pixel depth.

16. A method of processing an image containing pixels, comprising:

finding an intersection image within the image using an individual pixel property;

determining a boundary of a foreground region using the intersection image; and

filling in the foreground region within the boundary to segment the foreground region from the image.

17. The method of claim 16, wherein determining a boundary further comprises calculating a histogram for each pixel in the intersection image.

18. The method of claim 17, wherein filling in the foreground region further comprises backprojecting the histogram onto the boundary.

19. The method of claim 18, wherein backprojecting further comprises using connectivity to determine which pixels to include within the foreground region.

20. The method of claim 19, wherein using connectivity comprises comparing a histogram value of each pixel within the intersection image to a threshold and including the pixel within the foreground region is the histogram value is greater than the histogram threshold.