Visual conditioning for augmented-reality-assisted video conferencing

• US 9,154,732 B2
• Filed: 12/31/2012
• Issued: 10/06/2015
• Est. Priority Date: 04/09/2012
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Guleryuz, Onur G., Kalker, Antonius
• Primary Examiner(s): ZENATI, AMAL S
• Application Number:
  US13/732,123
• Publication Number:
  US 20130265382A1
• Time in Patent Office: 1,009 Days
• Field of Search: 348/14.08, 382/118, 382/284
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

An apparatus comprising a processor configured to receive a video teleconferencing image, wherein the video teleconferencing image has a background and at least one foreground, wherein the foreground has a boundary, determine the boundary of the foreground, overlay the boundary of the foreground with a drawing area, wherein the drawing area obscures the boundary, and replace the background of the image with a new background.

Claims

1. An apparatus comprising:

a receiver configured to receive a video teleconferencing image, wherein the video teleconferencing image comprises an original background and at least one foreground; and

a processor coupled to the receiver, wherein the processor is configured to;

determine an estimated boundary between the foreground and the original background, wherein the boundary comprises a plurality of pixels;

determine a drawing area to obscure segmentation errors associated with the boundary, wherein determining the drawing area comprises;

establishing a contour following flow field comprising a directional neighborhood around each boundary pixel to indicate directions following a contour of the boundary;

estimating a spread of the segmentation errors in directions indicated by the contour following flow field; and

deriving the drawing area to encompass the segmentation errors utilizing the contour following flow field and the estimated segmentation error spread;

overlay the boundary with the drawing area; and

replace the original background of the video teleconferencing image with a new background.

2. The apparatus of claim 1, wherein the drawing area contains one or more textures, and wherein the textures of the texturized drawing area are selected from a group consisting of:

random textures, semi-random textures, patterned textures, and pattern fields selected to match a local structure of the foreground.

3. The apparatus of claim 1, wherein the processor is further configured to select the new background from a plurality of new backgrounds, and wherein the selection includes harmonizing a color of the new background with a color of the foreground.

4. The apparatus of claim 1, wherein the processor is further configured to:

receive a second video teleconferencing image, wherein the second video teleconferencing image comprises an original background and a foreground;

determine an estimated boundary between the original background and the foreground of the second video teleconferencing image;

determine a drawing area of the second video teleconferencing image to obscure segmentation errors associated with the boundary of the second video teleconferencing image;

overlay the boundary of the second video teleconferencing image with the drawing area, wherein the drawing area of the second video teleconferencing image obscures the boundary of the second video teleconferencing image; and

replace the original background of the second video teleconferencing image with a new background, wherein the new background of the second video teleconferencing image is distinct from the new background of the video teleconferencing image in color, texture, or both.

5. The apparatus of claim 1, wherein the processor is further configured to select the new background from a plurality of new backgrounds, and wherein the selection includes selecting a texture to facilitate a smooth visual transition from the foreground to the new background.

6. The apparatus of claim 1, wherein the contour following flow field is a linear fit to the boundary, and wherein the segmentation error spread is estimated in both a direction of the contour following flow field at each boundary pixel and a direction orthogonal to the contour following flow field at each boundary pixel.

7. The apparatus of claim 1, wherein the processor is further configured to:

select at least one dominant foreground color from the foreground; and

select the new background by determining that the new background comprises at least one dominant background color matching the dominant foreground color.

8. The apparatus of claim 7, wherein selecting the dominant foreground color is accomplished by passing at least a portion of the video teleconferencing image through a first and second low pass filter, and wherein the first low pass filter has a lower pass band frequency than the second low pass filter.

9. The apparatus of claim 7, wherein the processor is further configured to modulate the new background over time to form a time varying background.

10. The apparatus of claim 9, wherein the modulation is accomplished using base background frames formed from pre-existing video.

11. The apparatus of claim 7, wherein the foreground comprises two dominant foreground colors, wherein the new background comprises two dominant background colors, and wherein the first dominant background color matches the first dominant foreground color and the second dominant background color matches the second dominant foreground color.

12. The apparatus of claim 1, wherein the processor is further configured to:

design one or more textures to be rendered in the drawing area;

select the new background; and

combine the foreground, the one or more textures, the drawing area, and the new background to form a composite image.

13. The apparatus of claim 12, wherein selecting the new background comprises:

averaging pixel colors in the original background to obtain an average original background color; and

determining a color of the new background based on the average original background color, wherein the color of the new background is analogous or split-analogous in color scheme to the average original background color.

14. The apparatus of claim 12, wherein designing one or more textures comprises approximating a local structure of the foreground using textures designed using one or more techniques selected from a group consisting of:

random field generation, semi-random field generation, patterned field generation, auto-regressive moving average processes, and texture-resampling processes.

15. A method implemented in a network element, the method comprising:

receiving a video teleconferencing image, wherein the video teleconferencing image comprises an original background and at least one foreground;

determining an estimated boundary between the foreground and the original background, wherein the boundary comprises a plurality of pixels;

determining a drawing area to obscure segmentation errors associated with the boundary, wherein determining the drawing area comprises;

establishing a contour following flow field comprising a directional neighborhood around each boundary pixel to indicate directions following a contour of the boundary;

estimating a spread of the segmentation errors in directions indicated by the contour following flow field; and

deriving the drawing area to encompass the segmentation errors utilizing the contour following flow field and the estimated segmentation error spread;

overlaying the boundary with the drawing area; and

replacing the original background of the image with a new background.

16. The method of claim 15, further comprising selecting the new background from a plurality of new backgrounds, wherein the selection includes harmonizing a color of the new background with a color of the foreground.

17. The method of claim 15, wherein the contour following flow field is a linear fit to the boundary, and wherein the segmentation error spread is estimated in both a direction of the contour following flow field at each boundary pixel and a direction orthogonal to the contour following flow field at each boundary pixel.

18. The method of claim 15, further comprising:

selecting at least one dominant foreground color from the foreground; and

select the new background by determining that the new background comprises at least one dominant background color matching the dominant foreground color.

19. The method of claim 18, wherein selecting the dominant foreground color is accomplished by passing at least a portion of the video teleconferencing image through a first and second low pass filter, and wherein the first low pass filter has a lower pass band frequency than the second low pass filter.

20. The method of claim 15, further comprising selecting the new background by:

averaging pixel colors in the original background to obtain an average original background color; and

determining a color of the new background based on the average original background color,

wherein the color of the new background is analogous or split-analogous in color scheme to the average original background color.