Multimodal Foreground Background Segmentation

• US 20140307056A1
• Filed: 06/14/2013
• Published: 10/16/2014
• Est. Priority Date: 04/15/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kirk, Adam G., Collet Romea, Alvaro, Zhang, Bao
• Application Number:
  US13/918,747
• Publication Number:
  US 20140307056A1
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/596 from three or more stereo images
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/75 : Chroma key

Abstract

The subject disclosure is directed towards a framework that is configured to allow different background-foreground segmentation modalities to contribute towards segmentation. In one aspect, pixels are processed based upon RGB background separation, chroma keying, IR background separation, current depth versus background depth and current depth versus threshold background depth modalities. Each modality may contribute as a factor that the framework combines to determine a probability as to whether a pixel is foreground or background. The probabilities are fed into a global segmentation framework to obtain a segmented image.

Claims

1. A system comprising, a foreground background segmentation framework, including a multimodal segmentation algorithm configured to accept contribution factors from different segmentation modalities and process the contribution factors to determine foreground versus background data for each element of an image that is useable to determine whether that element is a foreground or background element.

2. The system of claim 1 wherein at least one element comprises a pixel.

3. The system of claim 1 wherein the foreground versus background data comprises a probability score.

4. The system of claim 1 wherein the different segmentation modalities correspond to any of:

a red, green blue (RGB) background subtraction, chroma keying, infrared (IR) background subtraction, a current computed depth versus previously computed background depth evaluation, or a current depth versus threshold depth evaluation.

5. The system of claim 1 wherein the foreground background segmentation framework is further configured to output the foreground versus background data for each element to a global binary segmentation algorithm.

6. The system of claim 1 wherein the framework is configured to apply a weight for each contribution factor.

7. The system of claim 6 wherein the framework is configured to select a weight set from among a plurality of weight sets to apply the weight for each contribution factor.

8. The system of claim 6 wherein the framework is coupled to a multiple camera set environment, and wherein the framework is configured to apply a weight set to one camera set that is different from a weight set applied to another camera set.

9. The system of claim 1 wherein the framework is coupled to a multiple camera set environment, and wherein the framework is configured to determine the foreground versus background data based on zero or more contribution factors in conjunction with information that corresponds to other camera foreground versus background data.

10. The system of claim 1 wherein the framework is configured to determine the foreground versus background data based on zero or more contribution factors and detection information processed from an image.

11. A method, comprising, processing a frame of image data and processing depth data computed from a corresponding depth-related image, including performing background subtraction on an element of the image data to obtain a background subtraction contribution factor for that element, determining one or more other depth-based contribution factors based upon the depth data associated with that element, computing a combined data term based at least in part upon a contribution from the background contribution factor and a contribution from each of the one or more other depth-based contribution factors, and using the data term in conjunction with other data terms as input to a global binary segmentation mechanism to obtain a segmented image.

12. The method of claim 11 further comprising processing a frame of image data using chroma keying to obtain a chroma keying contribution factor, for the element and wherein computing the combined data term further comprises using a contribution from the chroma keying contribution factor.

13. The method of claim 11 wherein performing the background subtraction on an element of the image data comprises performing infrared background subtraction using captured infrared image data for a current element and previously captured background infrared image data.

14. The method of claim 11 wherein determining the one or more other depth-based contribution factors comprises evaluating a difference between currently captured depth data corresponding to the element and previously captured background depth data corresponding to the element.

15. The method of claim 11 wherein determining the one or more other depth-based contribution factors comprises evaluating currently captured depth data corresponding to the element and threshold depth data

16. The method of claim 11 further comprising, using information corresponding to background versus foreground information corresponding to at least one other cameras as in computing the combined data term.

17. One or more machine-readable storage media or logic having executable instructions, which when executed perform steps, comprising:

(a) selecting a pixel as a selected pixel;

(b) processing pixel data, including;

processing red, green and blue (RGB) pixel data of one or more images to determine one or more RGB contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in a current image;

processing infrared (IR) pixel data of one or more infrared images to determine one or more IR contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in the current image;

processing pixel depth data to determine one or more depth-based contributing factors indicative of whether the selected pixel is likely a foreground or background pixel in the current image;

(c) combining the contributing factors into a data term for the selected pixel;

(d) maintaining the data term for the selected pixel independent of other data terms for any other pixels;

(e) selecting a different pixel as the selected pixel; and

(f) returning to step (b) for a plurality of pixels to obtain a plurality of data terms.

18. The one or more machine-readable storage media or logic of claim 17 wherein processing the RGB pixel data of the one or more images to determine the one or more RGB contributing factors comprises performing at least one of:

background subtraction based on a previous RGB background image and a current RGB image, or performing chroma keying based on known background data and a current RGB image.

19. The one or more machine-readable storage media or logic of claim 17 wherein processing the IR pixel data of the one or more images to determine the one or more IR contributing factors comprises performing background subtraction based on a previous IR background image and a current IR image.

20. The one or more machine-readable storage media or logic of claim 17 wherein processing the pixel depth data to determine the one or more depth-based contributing factors comprises performing at least one of:

evaluating current pixel depth data against previous background pixel data, or evaluating current pixel depth data against threshold depth data.