System for background subtraction with 3D camera

• US 9,530,044 B2
• Filed: 07/21/2015
• Issued: 12/27/2016
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Board of Trustees of The University of Illinois
• Original Assignee: Board of Trustees of The University of Illinois
• Inventors: Nguyen, Quang H, Do, Minh N, Patel, Sanjay J., Dabbelt, Daniel P., Lin, Dennis J.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US14/805,335
• Publication Number:
  US 20150356716A1
• Time in Patent Office: 525 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information into three different regions including foreground (FG), background (BG), and unclear (UC); categorizes UC pixels as FG or BG using a function that considers the color and background history (BGH) information associated with the UC pixels and the color and BGH information associated with pixels near the UC pixels; examines the pixels marked as FG and applies temporal and spatial filters to smooth boundaries of the FG regions; constructs a new image by overlaying the FG regions on top of a new background; displays a video feed of the new image in a display device; and continually maintains the BGH.

Claims

1. A system for background image subtraction, the system comprising:

a computing device having a processor and memory, the processor programmed to receive video data, via a 3D application programming interface (API), from a camera, the video data containing (i) at least one colored image of one or more subjects and (ii) corresponding depth information;

the processor further programmed to;

assign pixels of the video data to one of three different regions including foreground (FG), background (BG), and unclear (UC);

re-assign UC pixels to FG or BG using a function that considers (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; and

construct at least one new image by overlaying the FG regions on top of a new background.

2. The system of claim 1, wherein the function for re-assigning the UC pixels is based on (i) color dissimilarity between UC pixels and pixels near the UC pixels of the at least one colored image and (ii) color dissimilarity between the UC pixels and pixels near the UC pixels of the BGH information.

3. The system of claim 1, wherein the processor is further programmed to apply at least one of (i) spatial smoothing to a FG boundary by utilizing a median filter and (ii) temporal filtering by inspecting the color images and FG assignment of a current and previous frames.

4. The system of claim 1, wherein the new background comprises at least one image from a second video.

5. The system of claim 1, wherein assigning pixels of the video data to one of three different regions comprises, for each of a plurality of pixels at varying depths along a Z axis from the camera:

assigning the pixel as FG if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity;

assigning the pixel as UC if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

assigning all other pixels not categorized as FG or UC as BG.

6. The system of claim 1, wherein the processor is further programmed to:

detect and label as connected components pixels that are adjacent to each other, are in the same region, and have depth values smaller than a predetermined threshold, wherein re-assigning the UC pixels further comprises;

re-assigning as BG any FG connected component having a cross-sectional area less than a predetermined threshold area, γ

;

re-assigning as BG any UC connected component having a cross-sectional area greater than a second predetermined threshold area, γ

′

; and

re-assigning as BG any UC connected component having a cross-sectional area less than γ and

for which no adjacent component thereof includes an FG connected component having a cross-sectional area greater than γ

.

7. The system of claim 6, wherein the processor is further programmed to:

detect a FG fringe of FG pixels along boundaries using a morphological opening operator applied thereto;

merge the FG fringe into a UC region of remaining UC pixels;

compare each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not assigned as UC pixels;

re-assign each compared pixel in the UC region as either BG or FG based on the neighbor pixel that best matches the compared pixel; and

merge any remaining UC region with the FG regions.

8. The system of claim 7, wherein the processor is further programmed to, after re-assigning the compared UC pixels:

re-assign as BG the remaining UC pixels that lie under a center of mass of each FG component; and

re-assign as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region.

9. The system of claim 6, wherein the processor is further programmed to:

determine if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copy only those UC pixels with RGB values that are unchanged into an updated UC region; and

apply a median filter on the FG pixels to smoothen boundaries of the FG connected components.

10. The system of claim 7, wherein the camera includes a color sensor and a depth sensor, and the processor is further programmed to, before detecting and merging the FG fringe into the UC region:

warp the FG and UC regions from a depth sensor viewpoint to a color sensor viewpoint;

compare red/green/blue (RGB) values of the UC pixels with corresponding BGH information, and where a match is found, assign the UC pixel as a BG pixel; and

dilate any remaining UC pixels to surrounding pixels thereof that are not assigned as FG pixels.

11. The system of claim 10, wherein the processor is further programmed to:

compute an average depth value for each FG connected component; and

select between a near mode and a far mode based on the average depth of the largest FG connected component, where the near mode is selected if the average depth is less than a predetermined threshold depth.

12. The system of claim 11, wherein the processor is further programmed to, when in near mode, for each FG connected component, assign as BG a plurality of UC pixels that lie under a center of mass of the FG connected component before comparison thereof with the BGH information.

13. The system of claim 10, wherein the processor is further programmed to:

propagate the depth information for the UC region and for the FG connected components from the depth sensor into the color sensor, to synchronize the depth information with corresponding pixels in the color image when the color and depth sensors are respectively positioned at different locations in 3D space; and

for each of a plurality of the pixels having unknown RGB values;

determine if the pixel is surrounded by other FG or UC pixels within a predetermined support window;

re-assign the unknown pixels that are surrounded by FG pixels as FG; and

re-assign the unknown pixels that are surrounded by UC pixels as UC.

14. The system of claim 10, wherein the processor is further programmed to reset the BGH information in response to detecting at least one of:

an indication that the camera is shaken;

one or more subjects being too close to the camera;

a subject from the background moving into the foreground; and

a significant lighting change.

15. A computer-implemented method for background subtraction, the method executable by a computing device having at least one processor and memory, the at least one processor programmed to receive video data, via a 3D application programming interface (API), from a camera, the video data containing (i) at least one colored image of one or more subjects and (ii) corresponding depth information, the method comprising:

assigning, by the at least one processor, pixels of the video data to one of three different regions including foreground (FG), background (BG), and unclear (UC);

re-assigning, by the at least one processor, UC pixels to FG or BG using a function that considers (i) color and background history (BGH) information associated with the UC pixels and (ii) color and BGH information associated with pixels near the UC pixels; and

constructing, by the at least one processor, a new image using the FG pixels and a new background.

16. The method of claim 15, wherein the new background comprises at least one image from a second video.

17. The method of claim 15, further comprising applying, by the at least one processor, at least one of (i) spatial smoothing to a FG boundary by utilizing a median filter and (ii) temporal filtering by inspecting the color images and FG assignment of a current and previous frames.

18. The method of claim 15, wherein assigning pixels of the video data to one of three different regions comprises the processor, for each of a plurality of pixels at varying depths along a Z axis from the camera:

assigning the pixel as FG if a depth thereof is less than a predetermined threshold distance from the camera and an intensity thereof is greater than a predetermined threshold intensity;

assigning the pixel as UC if a depth thereof is less than the predetermined threshold distance and an intensity thereof is less than the predetermined threshold intensity; and

categorizing all other pixels not categorized as FG or UC as BG.

19. The method of claim 18, further comprising the processor:

detecting a FG fringe of FG pixels along colored edges using a morphological opening operator applied thereto;

merging the FG fringe into a UC region of remaining UC pixels;

comparing each of a plurality of pixels in the UC region with corresponding neighbor pixels that are not assigned UC pixels;

re-assigning each compared pixel in the UC region as either BG or FG based on the neighbor pixel that best matches the compared pixel; and

merging any remaining UC region with the FG regions.

20. The method of claim 19, further comprising the processor:

detecting and labeling as connected components pixels that are adjacent to each other, are in the same region, and have depth values smaller than a predetermined threshold;

re-assigning as BG the remaining UC pixels that lie under a center of mass of each FG component;

re-assigning as BG a plurality of smallest UC connected components based on a median filter applied to the remaining UC pixels in the UC region;

determining if a red/blue/green (RGB) value of each of a plurality of UC pixels is unchanged from a previous frame;

copying only those UC pixels with RGB values that are unchanged into an updated UC region; and

applying a median filter on the FG pixels to smoothen edges of the FG connected components.