System for background subtraction with 3D camera

• US 8,649,592 B2
• Filed: 08/30/2010
• Issued: 02/11/2014
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Board of Trustees of The University of Illinois, Human Med AG
• Original Assignee: University of Illinois
• Inventors: Nguyen, Quang Huu, Do, Minh Ngoc, Patel, Sanjay Jeram, Dabbelt, Daniel Palmer, Lin, Dennis Jinn
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US12/871,428
• Publication Number:
  US 20120051631A1
• Time in Patent Office: 1,261 Days
• Field of Search: 382/154, 382/285, 345419-427, 356 12- 14, 348 42- 60
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information into three different regions including foreground (FG), background (BG), and unclear (UC); categorizes UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels; examines the pixels marked as FG and applies temporal and spatial filters to smooth boundaries of the FG regions; constructs a new image by overlaying the FG regions on top of a new background; displays a video feed of the new image in a display device; and continually maintains the BGH.

Claims

1. A system for background image subtraction comprising:

a computing device having a processor and memory and coupled with a three-dimensional (3D) video camera, the processor programmed to receive a video feed from the camera, the video feed containing colored images of one or more subject that include depth information;

the processor further programmed, for a colored image, to;

segment pixels and corresponding depth information of the images into three different regions including foreground (FG), background (BG), and unclear (UC);

categorize UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels;

examine the pixels marked as FG and apply temporal and spatial filters to smooth boundaries of the FG regions;

construct a new image by overlaying the FG regions on top of a new background;

display a video feed of the new image in a display device coupled with the computing device; and

continually maintain the BGH.

2. The system of claim 1, where the function for categorizing the UC pixels is based on color dissimilarity between UC pixels and neighbor pixels of the colored image and based on color dissimilarity between the UC pixels and neighbor pixels of the BGH.

3. The system of claim 1, where the spatial smoothing function for the FG boundary is executed with a median filter and temporal filtering is achieved by inspecting the color images and FG categorization of the current and previous frames, respectively.

4. The system of claim 1, where the new background comprises a second video feed.

5. The system of claim 1, where segmenting further comprises the processor programmed to, for each of a plurality of pixels at varying depths along a Z axis from the camera:

categorize the pixel as foreground (FG) if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity;

categorize the pixel as unclear (UC) if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorize all other pixels not categorized as FG or UC as background (BG).

6. The system of claim 1, where the processor is further programmed to:

detect and label pixels as region-specific connected components that are adjacent to each other, in the same region, and have depth values smaller than a predetermined threshold;

categorize as BG any FG connected component having a cross-sectional area less than a predetermined threshold area, y;

categorize as BG any UC connected component having a cross-sectional area greater than y; and

categorize as BG any UC connected component having a cross-sectional area less than y and for which no adjacent component thereof includes a FG connected component having a cross-sectional area greater than y.

7. The system of claim 6, where the processor is further programmed to:

detect a FG fringe of FG pixels along the boundaries using a morphological opening operator applied thereto;

merge the FG fringe into a UC region of remaining UC pixels;

compare each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not categorized as UC pixels;

categorize each compared pixel in the UC region as either BG or FG based on the neighbor pixels that best match the compared pixel; and

merge any remaining UC region with the FG regions.

8. The system of claim 7, where, after categorization of the compared UC pixels, the processor further programmed to:

categorize as BG the remaining UC pixels that lie under a center of mass of each FG component; and

categorize as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region.

9. The system of claim 8, where the processor is further programmed to:

determine if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copy only those UC pixels values that are unchanged into an updated UC region; and

apply a median filter on the FG pixels to smoothen the boundaries of the FG connected components.

10. The system of claim 7, where the camera includes a color sensor and a depth sensor, and before detecting and merging the FG fringe into the UC region, the processor is further programmed to:

warp the FG and UC regions from a depth sensor viewpoint to a color sensor viewpoint;

compare red/green/blue (RGB) values of the UC pixels with a corresponding BGH thereof, and where a match is found, assign the UC pixel as a BG pixel; and

dilate any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels.

11. The system of claim 10, where the processor is further programmed to:

compute an average depth value for each FG component; and

select between a near mode and a far mode based on the average depth of the largest FG connected component, where the near mode is selected if the average depth is less than a predetermined threshold depth.

12. The system of claim 11, when in near mode, for each FG component, the processor further programmed to:

assign as BG a plurality of UC pixels that lie under a center of mass of corresponding one or more FG component before comparison thereof with the BGH.

13. The system of claim 10, where the processor is further programmed to:

propagate the depth information for the UC region and for the FG connected components from the depth sensor into the color sensor, to synchronize the depth information with corresponding pixels in the color image when the color and depth sensors are positioned at a different location in the 3D space;

for each of a plurality of the pixels having unknown RGB values, determine if the pixel is surrounded by other FG or UC pixels within a predetermined support window; and

categorize the unknown pixels as FG that are surrounded by FG pixels and as UC that are surrounded by UC pixels.

14. The system of claim 10, where the processor is further configured to reset the BGH given the occurrence of one selected from the group consisting of:

receive an indication that the camera is shaken;

one of the one or more subject is too close to the camera;

a subject in the background moves to the foreground; and

in the presence of a significant lighting change.

15. A computer-implemented method for background subtraction, the method executable by a computing device having a processor and memory, the processor programmed to receive a video feed from the camera, the video feed containing colored images of one or more subject that include depth information, the method comprising:

segmenting, by the processor, pixels and corresponding depth information of the images into three different regions including foreground (FG), background (BG), and unclear (UC);

categorizing, by the processor, UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels;

deleting, by the processor, the BG region up to and around colored edges of the FG region;

applying, by the processor, temporal and spatial filters to smooth the colored edges of the FG regions;

constructing, by the processor, a new image by overlaying the FG regions on top of a new background;

displaying, by the processor, the new image in a display device coupled with the computing device; and

continually maintaining the BGH by the processor.

16. The method of claim 15, where the new background comprises a second video feed and where the new image is displayed in a third video feed.

17. The method of claim 15, where the spatial smoothing function for the FG colored edges is executed with a median filter and temporal filtering is achieved by inspecting the color images and FG categorization of the current and previous frames, respectively.

18. The method of claim 15, where segmenting further comprises the processor, for each of a plurality of pixels at varying depths along a Z axis from the camera:

categorizing the pixel as foreground (FG) if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity;

categorizing the pixel as unclear (UC) if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorizing all other pixels not categorized as FG or UC as background (BG).

19. The method of claim 18, the method further comprising the processor:

detecting a FG fringe of FG pixels along the colored edges using a morphological opening operator applied thereto;

merging the FG fringe into a UC region of remaining UC pixels;

comparing each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not categorized as UC pixels;

categorizing each compared pixel in the UC region as either BG or FG based on the neighbor pixels that best match the compared pixel; and

merging any remaining UC region with the FG regions.

20. The method of claim 19, the method further comprising the processor:

categorizing as BG the remaining UC pixels that lie under a center of mass of each FG component;

categorizing as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region.

determining if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copying only those UC pixels values that are unchanged into an updated UC region; and

applying a median filter on the FG pixels to smoothen the edges of the FG connected components.

21. The method of claim 19, where the camera includes a color sensor and a depth sensor, and before detecting and merging the FG fringe into the UC region, the method further comprising the processor:

warping the FG and UC regions from a depth sensor viewpoint to a color sensor viewpoint;

comparing red/green/blue (RGB) values of the UC pixels with a corresponding BGH thereof, and where a match is found, assign the UC pixel as a BG pixel;

dilating any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels; and

computing an average depth value for each FG component.

22. The method of claim 21, the method further comprising the processor resetting the BGH given the occurrence of one selected from the group consisting of:

receiving an indication that the camera is shaken;

one of the one or more subject is too close to the camera;

a subject in the background moves to the foreground; and

in the presence of a significant lighting change.

23. A non-transitory computer-readable storage medium comprising a set of instructions for background image subtraction executable by a computing device having a processor and memory, the processor programmed to receive a video feed from the camera, the video feed containing colored images of one or more subject that include depth information, the computer-readable medium comprising:

instructions to direct the processor to segment pixels and corresponding depth information of the images into three different regions including foreground (FG), background (BG), and unclear (UC);

instructions to direct the processor to categorize UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels;

instructions to direct the processor to examine the pixels marked as FG and apply temporal and spatial filters to smooth boundaries of the FG regions;

instructions to direct the processor to construct a new image by overlaying the FG regions on top of a new background;

instructions to direct the processor to display the new image in a display device coupled with the computing device; and

instructions to direct the processor to continually maintain the BGH.

24. The non-transitory computer-readable storage medium of claim 23, where the new background comprises a second video feed, further comprising:

instructions to direct the processor to display the new image in a third video feed.

25. The non-transitory computer-readable storage medium of claim 23, further comprising:

instructions to direct the processor to execute the spatial smoothing function for the FG boundaries with a median filter and temporal filtering is achieved by inspecting the color images and FG categorization of the current and previous frames, respectively.

26. The non-transitory computer-readable storage medium of claim 23, further comprising instructions to direct the processor to, for each of a plurality of pixels at varying depths along a Z axis from the camera:

categorize the pixel as foreground (FG) if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity;

categorize the pixel as unclear (UC) if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorize all other pixels not categorized as FG or UC as background (BG).

27. The non-transitory computer-readable storage medium of claim 26, further comprising instructions to direct the processor to:

detect a FG fringe of FG pixels along the boundaries using a morphological opening operator applied thereto;

merge the FG fringe into a UC region of remaining UC pixels;

compare each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not categorized as UC pixels;

categorize each compared pixel in the UC region as either BG or FG based on the neighbor pixels that best match the compared pixel; and

merge any remaining UC region with the FG regions.

28. The non-transitory computer-readable storage medium of claim 27, further comprising instructions to direct the processor to:

categorize as BG the remaining UC pixels that lie under a center of mass of each FG component;

categorize as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region.

determine if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copy only those UC pixels values that are unchanged into an updated UC region; and

apply a median filter on the FG pixels to smoothen the boundaries of the FG connected components.

29. The non-transitory computer-readable storage medium of claim 27, where the camera includes a color sensor and a depth sensor, and before detecting and merging the FG fringe into the UC region, further comprising instructions to direct the processor to:

warp the FG and UC regions from a depth sensor viewpoint to a color sensor viewpoint;

compare red/green/blue (RGB) values of the UC pixels with a corresponding BGH thereof, and where a match is found, assign the UC pixel as a BG pixel;

dilate any remaining UC pixels to surrounding pixels thereof that are not categorized as FG pixels; and

compute an average depth value for each FG component.

30. The non-transitory computer-readable storage medium of claim 27, further comprising instructions to direct the processor to reset the BGH given the occurrence of one selected from the group consisting of:

receiving an indication that the camera is shaken;

one of the one or more subject is too close to the camera;

a subject in the background moves to the foreground; and

in the presence of a significant lighting change.