SYSTEM AND METHOD FOR FEATURE LEVEL FOREGROUND SEGMENTATION

• US 20080181499A1
• Filed: 01/31/2007
• Published: 07/31/2008
• Est. Priority Date: 01/31/2007
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Yang, Tao, Kimber, Donald G., Chen, Francine R.
• Granted Patent: US 7,916,944 B2
• Time in Patent Office: Days
• US Class Current: 382/174
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/155 involving morphological operators
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

Foreground segmentation in real world dynamic scenes, including under various lighting and shadow conditions, is disclosed. It may be used with one or multiple cameras for various automated tasks, such as classification of moving object, tracking moving objects, and event detection in various indoor or outdoor environments. Pixel to pixel subtraction is performed on each frame, followed by a feature-level based foreground segmentation to properly validate the foreground pixels. In this step, for each pixel in the image, a neighborhood of pixels is selected, and the aggregate change in the neighborhood image is used to classify foreground and background pixels. Normalized cross correlation is then applied to the neighborhood of each pixel that was confirmed to be foreground pixel.

Claims

1. A method for performing feature level segmentation, comprising:

a. selecting a background image;

b. selecting an input image;

c. for each input pixel of the input image;

c.i. classifying the pixel as a candidate foreground pixel or candidate background pixel;

c.ii. selecting a neighborhood image about the pixel;

c.iii. determining the aggregate change in the neighborhood image;

c.iv. depending upon the aggregate change, reclassifying the pixel as a foreground or a background pixel;

d. outputting an indication of all of the image pixels reclassified as foreground pixels.

2. The method of claim 1, further comprising the steps:

e. for each pixel reclassified as foreground pixel;

e.i. computing a difference of the neighborhood image of the reclassified foreground pixel and the corresponding background neighborhood image; and

,

e.ii. when the difference passes a set threshold, outputting an indication that the pixel is a confirmed foreground pixel.

3. The method of claim 2, further comprising the step:

e.iii. generating a feature extracted image from all of the confirmed foreground pixels.

4. The method of claim 1, further comprising the step:

e.iii. generating a feature extracted image from all of the input pixels that were reclassified as foreground pixels.

5. The method of claim 1, wherein determining the aggregate change in step c.iii. comprises counting the total number of pixels in the neighborhood image that were classified as candidate foreground pixels in step c.i.

6. The method of claim 5, wherein determining the aggregate change in step c.iii. further comprises comparing the total number to a threshold.

7. The method of claim 1, wherein determining the aggregate change in step c.iii. comprises calculating the percentage of pixels in the neighborhood image that were classified as candidate foreground pixels in step c.i.

8. The method of claim 7, wherein determining the aggregate change in step c.iii. further comprises comparing the percentage to a threshold.

9. The method of claim 1, wherein determining the aggregate change in step c.iii. comprises calculating the total intensity of pixels that were classified as candidate foreground pixels in step c.i.

10. The method of claim 9, wherein determining the aggregate change in step c.iii. further comprises subtracting the total intensity from a total background intensity of corresponding background neighborhood pixels to generate intensity difference, and comparing an absolute of the intensity difference to a threshold.

11. The method of claim 2, wherein said computing a difference of step e.i. comprises computing normalized cross correlation for the neighborhood image and the background neighborhood image.

12. The method of claim 11, wherein the cross-correlation is computed using an integral image.

13. The method of claim 2, wherein said computing a difference of step e.i. comprises constructing neighborhood image intensity matrix and background neighborhood image intensity matrix and performing a dot product of the neighborhood image intensity matrix and the background neighborhood image intensity matrix.

14. The method of claim 2, wherein said computing a difference of step e.i. comprises constructing neighborhood image gradient distribution and background neighborhood gradient distribution and computing the similarities of the neighborhood image gradient distribution and the background neighborhood image gradient distribution.

15. The method of claim 1, wherein step c.i. comprises comparing the input pixel to corresponding background pixel in the background image to classify the pixel as a candidate foreground pixel or candidate background pixel.

16. The method of claim 1, wherein step c.ii. comprises selecting a plurality of pixels about the pixel to define the neighborhood.

17. The method of claim 1, further comprising step b.i. updating the background image using the input image.

18. The method of claim 1, wherein step c.iv. comprises comparing the aggregate change to a threshold.

19. A system for performing feature level segmentation, comprising:

a video storage module receiving at least one stream of video from at least one video camera;

a feature segmented storage module storing feature segmented images;

a feature extraction module, said feature extraction module comprising;

a background image storage module storing a background image;

a pixel level subtraction module receiving images of the video stream and for each image classifying each pixel of the image as background or foreground pixel;

a foreground density estimation module receiving the background and foreground pixels from the pixel level subtraction module and for each pixel selecting a plurality of neighboring pixels to define a neighborhood image and calculating a total change of the neighborhood, based upon the total change said foreground density estimation module reclassifies each pixel as reclassified-background or reclassified-foreground pixel;

a feature extraction module receiving the reclassified-foreground pixels and generating feature segmented images and output the feature segmented images to be stored in the feature extraction module.

20. The system of claim 19, wherein the feature extraction module comprises a feature level subtraction module receiving the reclassified-foreground pixels and performing a normalized cross correlation on a neighborhood of each the reclassified-foreground pixel to select pixels for generating the feature extracted images.

21. The system of claim 20, further comprising pixel level background modeling module receiving the video stream and updating the background image.