System for background subtraction with 3D camera

• US 10,325,360 B2
• Filed: 10/16/2017
• Issued: 06/18/2019
• Est. Priority Date: 08/30/2010
• Status: Active Grant

• Current Assignee: Board of Trustees of The University of Illinois
• Original Assignee: Board of Trustees of The University of Illinois
• Inventors: Nguyen, Quang H., Do, Minh N., Patel, Sanjay J, Dabbelt, Daniel P., Lin, Dennis J.
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US15/785,407
• Publication Number:
  US 20180089814A1
• Time in Patent Office: 610 Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A system for background image subtraction includes a computing device coupled with a 3D video camera, a processor of the device programmed to receive a video feed from the camera containing images of one or more subject that include depth information. The processor, for an image: segments pixels and corresponding depth information to identify foreground regions associated with a target user.

Claims

1. A mobile computing device comprising:

at least one color sensor configured to obtain color frames of a scene having a target subject, the color frames having a color sensor viewpoint;

a depth sensor having an infrared illuminator and an infrared sensor, configured to generate depth data of the scene from a depth sensor viewpoint;

a processor and memory configured to;

a) identify a set of potential foreground pixel locations by processing the depth data;

b) detect one or more foreground connected components;

c) determine a near mode based on an average depth of the one or more foreground connected components;

d) responsive to determining the near mode, form a reduced set of potential foreground pixel locations by removing some pixel locations from the set of potential foreground pixel locations that are below a center of the one or more connected foreground components;

e) process the reduced set of potential foreground pixel locations to generate aligned foreground depth data associated with a target user by warping the reduced set of potential foreground pixel locations from the depth sensor viewpoint to the color sensor viewpoint; and

,

f) construct at least one new color image based on the aligned foreground depth data and the color frames from the at least one color sensor.

2. The mobile computing device of claim 1, wherein the processor and memory are further configured to display the at least one new color image as frames in a video stream.

3. The mobile computing device of claim 2, wherein the video stream is displayed in real-time.

4. The mobile computing device of claim 1, wherein identifying a set of potential foreground pixel locations by processing the depth data comprises, for each of a plurality of pixels, each pixel having a depth along a Z axis from the depth sensor, identifying a pixel as a potential foreground pixel if its respective depth is less than a predetermined threshold distance.

5. The mobile computing device of claim 1, wherein the near mode is determined according to an average depth of a largest connected foreground component.

6. The mobile computing device of claim 1, wherein the set of potential foreground pixels is further reduced according to color information and background history (BGH) information associated with the potential foreground pixels.

7. The mobile computing device of claim 1, further comprising, assigning depth values to each foreground pixel of the color frame having unknown depth values by interpolating the aligned foreground depth data.

8. The mobile computing device of claim 1, wherein detecting one or more foreground connected components comprises labeling as foreground connected components pixels that are (i) adjacent to each other, (ii) identified as potential foreground pixels, and (iii) have adjacent depth values within a predetermined threshold.

9. The system of claim 8, wherein foreground connected components are further limited to pixels having a depth value less than a predetermined threshold distance and an intensity value greater than a predetermined threshold intensity.

10. The system of claim 1, wherein the at least one new color image further comprises at least one image from a second video.