Background blurring for video conferencing

• US 7,783,075 B2
• Filed: 06/07/2006
• Issued: 08/24/2010
• Est. Priority Date: 06/07/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Zhang, Cha, He, Li-wei
• Primary Examiner(s): Ahmed; Samir A.
• Assistant Examiner(s): RASHIDIAN, MOHAMMAD M
• Application Number:
  US11/422,693
• Publication Number:
  US 20070286520A1
• Time in Patent Office: 1,539 Days
• Field of Search: 382/100, 382/103, 382/173, 382/264, 382/243, 382/181, 382/224, 382/118, 382/154, 351/208, 348/169, 348/14.07, 342/90, 703/6, 702/77, 358/500, 345/418, 345/597
• US Class Current: 382/103
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q

Abstract

Background blurring is an effective way to both preserve privacy and keep communication effective during video conferencing. The present image background blurring technique is a light weight real-time technique to perform background blurring using a fast background modeling procedure combined with an object (e.g., face) detector/tracker. A soft decision is made at each pixel whether it belongs to the foreground or the background based on multiple vision features. The classification results are mapped to a per-pixel blurring radius image to blur the background. In another embodiment, the image background blurring technique blurs the background of the image without using the object detector.

Claims

1. A computer-implemented process for blurring the background in an image of an image sequence, comprising using a computer to perform the process actions of:

(a) dividing an image of an image sequence into pixels;

(b) creating a first probability map of each pixel'"'"'s probability of being a background pixel;

(c) using an object detector to find an object in the image and using any found object to determine the probability of each pixel being background thereby creating a second probability map;

(d) specifying that each pixel in the first probability map is not background, if it was determined by the object detector to belong to an object;

(e) combining the first and second probability maps to obtain a combined probability map that defines a probability of each pixel in the image being a background pixel;

(f) determining a blurring radius for each pixel based on its probability of being a background pixel; and

(g) blurring each pixel in the image using the blurring radius for each pixel to create an output image with a blurred background.

2. The computer-implemented process of claim 1 further comprising repeating process actions (a) through (g) for subsequent images in the image sequence.

3. The computer-implemented process of claim 1 wherein the process action of creating the first probability map, comprises the process actions of:

inputting a first image of an image sequence;

designating all pixels in the first image as foreground;

for each pixel in the first image, computing a mean color and a variance;

inputting the next image in the image sequence;

for each pixel in the next image, determining the probability of it being a background pixel;

updating the mean color and variance, using the probability of each pixel in said next image being a background pixel, to compute a running mean and variance for each pixel;

for each pixel, determining if the running variance is large;

if the running pixel variance is large, classifying the pixel as not being a background pixel;

if the running pixel variance is not large, classifying the pixel as being a background pixel; and

creating a probability map that each pixel is background based on the pixel classifications.

4. The computer-implemented process of claim 3 wherein the process action of determining the probability of a pixel belonging to the background is computed using a standard Gaussian kernel.

5. The computer-implemented process of claim 3 wherein each pixel in the input image is converted from red, green, blue (RGB) color space to YUV color space prior to computing a mean color and variance.

6. The computer-implemented process of claim 1 wherein the process action of using an object detector to find an object in the image and using any found object to determine the probability of a pixel being background thereby creating a second probability map, comprises the process actions of:

(a) inputting the image divided into pixels;

(b) using an object detector to detect any objects in the image;

if an object is detected,

verifying the location of a cropped sub-image of the object;

if the location of cropped sub-image of the object has not been verified for a number of tries, attempting to use the object detector to find the object in the whole image;

if the object is not found, designating all pixels as background,

if the object is found, designating each pixel in the image belonging to the object as having a 0 probability of being background;

if an object is not detected,

designating all pixels as background; and

(c) creating the second probability map by using the designation for each pixel.

7. The computer-implemented process of claim 1 wherein the object detector is a face detector and the object detected is a face.

8. The computer-implemented process of claim 1 wherein the process action of blurring each pixel in the image using the blurring radius for each pixel to create an output image with a blurred background, comprises the process actions of:

obtaining a combined probability derived from the first and second probability maps of each pixel in the image being background;

determining a blurring radius for each pixel based on the probability of it being background obtained from the combined probability map;

using the second probability map created by using the object detector to specify that pixels that belong to objects are not background; and

using the blurring radius to blur pixels that are background.

9. The computer-implemented process of claim 8 further comprising the process action of smoothing the image after the blurring radius for each image has been determined.

10. The computer-implemented process of claim 1 wherein the blurring radius is larger if the probability is higher that the pixel is background.

11. The computer-implemented process of claim 1 wherein the background in the image is blurred by using the corresponding blurring radius for each pixel and averaging the pixel colors within the radius.

12. The computer-implemented process of claim 1 wherein a Gaussian distribution with a rectangular kernel is used in determining each blurring radius, and wherein integral images are used in determining each blurring radius.

13. A system for blurring the background in an image, comprising:

a general purpose computing device;

a computer program comprising program modules executable by the general purpose computing device, wherein the computing device is directed by the program modules of the computer program to,

divide an input image into pixels;

perform background modeling to determine a first probability map that each pixel in the image is background;

perform object detection to find an object and specify that pixels of any object detected are not background pixels to create a second probability map that each pixel in the image is background;

perform image background blurring using a combination of first and second probability maps that each pixel in the image is background to create an image with a blurred background.

14. The system of claim 13 wherein object detection is performed by a user manually segmenting an object from background in the input image.

15. The system of claim 13 wherein object detection is automatically performed by using an object detector.

16. The system of claim 13 wherein the program module to perform image background blurring comprises:

for each pixel, averaging colors of the pixels in an area corresponding to the probability that a pixel is background, and replacing the pixel color with the averaged color.