System for background subtraction with 3D camera

• US 9,087,229 B2
• Filed: 02/06/2014
• Issued: 07/21/2015
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Board of Trustees of The University of Illinois, University of Illinois
• Original Assignee: University of Illinois
• Inventors: Do, Minh N., Patel, Sanjay J., Lin, Dennis J., Nguyen, Quang H., Dabbeit, Daniel P.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/174,498
• Publication Number:
  US 20140294288A1
• Time in Patent Office: 530 Days
• Field of Search: 382/154, 382/285, 356 12- 14, 345419-427, 348 42- 60
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information into three different regions including foreground (FG), background (BG), and unclear (UC); categorizes UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels; examines the pixels marked as FG and applies temporal and spatial filters to smooth boundaries of the FG regions; constructs a new image by overlaying the FG regions on top of a new background; displays a video feed of the new image in a display device; and continually maintains the BGH.

Claims

1. A system for background image subtraction, the system comprising:

at least one processor and memory, the at least one processor programmed with one or more instructions, which when executed cause the at least one processor to;

receive a video feed, the video feed comprising at least one colored image of at least one subject and depth information;

categorize pixels of the at least one colored image as foreground (FG), background (BG), or unclear (UC);

re-categorize UC pixels as FG or BG using (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; and

construct a new image using the FG pixels and a new background.

2. The system of claim 1, wherein re-categorizing the UC pixels is based on (i) color dissimilarity between the UC pixels and pixels near the UC pixels of the at least one colored image and (ii) color dissimilarity between the UC pixels and pixels near the UC pixels of the BGH information.

3. The system of claim 1, the instructions further comprising applying spatial and temporal filters to smooth boundaries of FG regions of FG pixels, wherein the spatial filter comprises a median filter and wherein applying the temporal filter comprises inspecting the at least one colored image and FG categorization of the current and previous frames, respectively.

4. The system of claim 1, wherein categorizing pixels comprises, for each of a plurality of pixels at varying depths along a Z axis:

categorizing the pixel as FG when a depth thereof is less than a predetermined threshold distance and an intensity thereof is greater than a predetermined threshold intensity;

categorizing the pixel as UC when a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorizing all other pixels not categorized as FG or UC as BG.

5. The system of claim 1, wherein the one or more instructions further cause the at least one processor to:

detect and label as connected components pixels that are adjacent to each other, are in the same region, and have depth values smaller than a predetermined threshold, wherein re-categorizing the UC pixels further comprises;

re-categorizing as BG any FG connected component having a cross-sectional area less than a predetermined threshold area, γ

;

re-categorizing as BG any UC connected component having a cross-sectional area greater than a second predetermined threshold area, γ

′

; and

re-categorizing as BG any UC connected component having a cross-sectional area less than γ and

for which no adjacent component thereof includes an FG connected component having a cross-sectional area greater than γ

.

6. The system of claim 5, wherein the one or more instructions further cause the at least one processor to:

detect an FG fringe of FG pixels along boundaries using a morphological opening operator applied thereto;

merge the FG fringe into a UC region of remaining UC pixels;

compare each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not categorized as UC pixels;

re-categorize each compared pixel in the UC region as either BG or FG based on the neighbor pixels that best match the compared pixel; and

merge any remaining UC region with the FG regions.

7. The system of claim 5, wherein, after re-categorizing the compared UC pixels, the one or more instructions further cause the at least one processor to:

re-categorize as BG the remaining UC pixels that lie under a center of mass of each FG connected component; and

re-categorize as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region.

8. The system of claim 5, wherein the one or more instructions further cause the at least one processor to:

determine if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copy only those UC pixel values that are unchanged into an updated UC region; and

apply a median filter on the FG pixels to smoothen the boundaries of the FG connected components.

9. The system of claim 6, further comprising a color sensor and a depth sensor, wherein the one or more instructions further cause the at least one processor to, before detecting and merging the FG fringe into the UC region:

warp the FG and UC regions from a depth sensor viewpoint to a color sensor viewpoint;

compare red/green/blue (RGB) values of the UC pixels with corresponding BGH information, and where a match is found, assign the UC pixel as a BG pixel; and

dilate any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels.

10. The system of claim 9, wherein the one or more instructions further cause the at least one processor to:

compute an average depth value for each FG connected component; and

select between a near mode and a far mode based on the average depth of the largest FG connected component, wherein the near mode is selected when the average depth is less than a predetermined threshold depth.

11. The system of claim 10, when in near mode, for each FG connected component, the one or more instructions further cause the at least one processor to:

assign as BG a plurality of UC pixels that lie under a corresponding center of mass of one or more FG connected components before comparison thereof with the BGH information.

12. The system of claim 9, wherein the one or more instructions further cause the at least one processor to:

propagate the depth information for the UC region and for the FG connected components from the depth sensor into the color sensor, to synchronize the depth information with corresponding pixels in the at least one color image when the color and depth sensors are positioned at a different location in 3D space;

for each of a plurality of the pixels having unknown RGB values,

determine if the pixel is surrounded by other FG or UC pixels within a predetermined support window; and

re-categorize the unknown pixels that are surrounded by FG pixels as FG and the unknown pixels that are surrounded by UC pixels as UC.

13. The system of claim 9, wherein the one or more instructions further cause the at least one processor to reset the BGH information upon receiving an indication that:

the video feed is from a sensor that has been shaken;

a subject is too close to a sensor;

a subject in the background moved to the foreground;

or

there was a lighting change.

28. The system of claim 1, the one or more instructions further causing the at least one processor to display a video feed of a series of new images in a display coupled to the processor.

29. The system of claim 1, the one or more instructions further causing the at least one processor to warp image information from a depth sensor point of view or a color sensor point of view to a desired point of view.

30. The system of claim 1, the one or more instructions further causing the at least one processor to track the BGH information.

14. A computer-implemented method for background subtraction, the method executable by a computing device having at least one processor and memory, the at least one processor programmed to receive a video feed, the video feed containing at least one colored image of at least one subject and depth information, the method comprising:

categorizing, by the at least one processor, pixels of the at least one colored image as foreground (FG), background (BG), or unclear (UC);

re-categorizing, by the processor, UC pixels as FG or BG using (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; and

constructing, by the processor, a new image using the FG pixels and a new background.

15. The method of claim 14, further comprising applying spatial and temporal filters to smooth boundaries of FG regions of FG pixels, wherein the spatial filter comprises a median filter and wherein applying the temporal filter comprises inspecting the at least one colored image and FG categorization of the current and previous frames, respectively.

16. The method of claim 14, wherein categorizing pixels comprises, for each of a plurality of pixels at varying depths along a Z axis:

categorizing the pixel as FG when a depth thereof is less than a predetermined threshold distance and an intensity thereof is greater than a predetermined threshold intensity;

categorizing the pixel as UC when a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorizing all other pixels not categorized as FG or UC as BG.

17. The method of claim 16, the method further comprising:

detecting an FG fringe of FG pixels along boundaries using a morphological opening operator applied thereto;

merging the FG fringe into a UC region of remaining UC pixels;

comparing each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not categorized as UC pixels;

re-categorizing each compared pixel in the UC region as either BG or FG based on the neighbor pixels that best match the compared pixel; and

merging any remaining UC region with the FG regions.

18. The method of claim 17, the method, further comprising:

detecting and labeling as connected components pixels that are adjacent to each other, are in the same region, and have depth values smaller than a predetermined threshold;

re-categorizing as BG the remaining UC pixels that lie under a center of mass of each FG connected component;

re-categorizing as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region;

determining if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copying only those UC pixel values that are unchanged into an updated UC region; and

applying a median filter on the FG pixels to smoothen the edges of the FG connected components.

19. The method of claim 18, further comprising, before detecting and merging the FG fringe into the UC region:

warping the FG and UC regions from a viewpoint of a depth sensor to a viewpoint of a color sensor;

comparing red/green/blue (RGB) values of the UC pixels with corresponding BGH information, and where a match is found, assigning the UC pixel as a BG pixel;

dilating any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels; and

computing an average depth value for each FG connected component.

20. The method of claim 19, the method further comprising resetting the BGH information upon receiving an indication that:

the video feed is from a sensor that has been shaken;

a subject is too close to a sensor;

a subject in the background moved to the foreground;

or

there was a lighting change.

21. A non-transitory computer-readable medium comprising a set of instructions for background image subtraction executable by a computing device having a processor and memory, the instructions causing the computing device to:

receive a video feed, the video feed containing at least one colored image of at least one subject and depth information;

categorize pixels of the at least one colored image as foreground (FG), background (BG), or unclear (UC);

re-categorize UC pixels as FG or BG using (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; and

construct a new image using the FG pixels and a new background.

22. The non-transitory computer-readable medium of claim 21, the instructions further causing the computing device to:

apply spatial and temporal filters to smooth boundaries of FG regions of FG pixels, wherein the spatial filter comprises a median filter and wherein applying the temporal filter comprises inspecting the at least one colored image and FG categorization of the current and previous frames, respectively.

23. The non-transitory computer-readable medium of claim 21, the instructions further causing the computing device, for each of a plurality of pixels at varying depths along a Z axis, to:

categorize the pixel as FG when a depth thereof is less than a predetermined threshold distance and an intensity thereof is greater than a predetermined threshold intensity;

categorize the pixel as UC when a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorize all other pixels not categorized as FG or UC as BG.

24. The non-transitory computer readable medium of claim 23, the instructions further causing the computing device to:

detect an FG fringe of FG pixels along boundaries using a morphological opening operator applied thereto;

merge the FG fringe into a UC region of remaining UC pixels;

compare each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not categorized as UC pixels;

re-categorize each compared pixel in the UC region as either BG or FG based on the neighbor pixels that best match the compared pixel; and

merge any remaining UC region with the FG regions.

25. The non-transitory computer-readable medium of claim 24, the instructions further causing the computing device to:

detect and label as connected components pixels that are adjacent to each other, are in the same region, and have depth values smaller than a predetermined threshold;

re-categorize as BG the remaining UC pixels that lie under a center of mass of each FG connected component;

re-categorize as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region;

determine if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copy only those UC pixel values that are unchanged into an updated UC region; and

apply a median filter on the FG pixels to smoothen the boundaries of the FG connected components.

26. The non-transitory computer-readable medium of claim 25, wherein the instructions further cause the computing device to, before detecting and merging the FG fringe into the UC region:

warp the FG and UC regions from a viewpoint of a depth sensor to a viewpoint of a color sensor;

compare red/green/blue (RGB) values of the UC pixels with corresponding BGH information, and where a match is found, assign the UC pixel as a BG pixel;

dilate any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels; and

compute an average depth value for each FG connected component.

27. The non-transitory computer-readable medium of claim 24, the instructions further causing the computing device to reset the BGH information upon receiving an indication that:

the video feed is from a sensor that has been shaken;

a subject is too close to a sensor,

a subject in the background moved to the foreground;

or

there was a significant lighting change.