Background modification in video conferencing

• US 10,116,901 B2
• Filed: 01/04/2016
• Issued: 10/30/2018
• Est. Priority Date: 03/18/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Avatar Merger Sub II, LLC
• Inventors: Shaburov, Victor, Monastyrshyn, Yurii
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US14/987,514
• Publication Number:
  US 20170019633A1
• Time in Patent Office: 1,030 Days
• Field of Search: 348 1401- 1416, 370259-271, 370351-357, 709201-207, 709217-248
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20124 Active shape model [ASM]
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/38 Registration of image sequences
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Methods and systems for real-time video processing can be used in video conferencing to modify image quality of background. One example method includes the steps of receiving a video including a sequence of images, identifying at least one object of interest (e.g., a face) in one or more of the images, detecting feature reference points of the at least one object of interest, and tracking the at least one object of interest in the video. The tracking may comprise aligning a virtual face mesh to the at least one object of interest in one or more of the images. Further, a background is identified in the images by separating the at least one object of interest from each image based on the virtual face mesh. The background is then modified in each of the images by blurring, changing a resolution, colors, or other parameters.

Claims

1. A computer-implemented method for real-time video processing, the method comprising:

receiving a video including a sequence of images;

identifying at least one object of interest in one or more of the images;

detecting at least one shape unit and a position vector of the at least one object of interest, and an intensity vector of the at least one shape unit, the at least one shape unit representing a parameter of a face of the at least one object of interest, and the position vector corresponding to a rotation around three axes and a translation along the axes;

tracking the at least one object of interest in the video, wherein the tracking is based on the at least one shape unit, the position vector, and the intensity vector;

identifying a background in each of the images by separating the at least one object of interest from each image based on the at least one shape unit, the position vector, and the intensity vector;

modifying the background in each of the images, thereby generating a modified background; and

generating a modified video which includes the at least one object of interest and the modified background.

2. The method of claim 1, wherein the modified background has a first image quality in the modified video and the at least one object of interest has a second image quality in the modified video;

and

wherein the first image quality is lower than the second image quality.

3. The method of claim 1, wherein the identifying of the background includes selecting an image portion which excludes pixels associated with a virtual face mesh.

4. The method of claim 1, wherein the modifying of the background includes blurring of the background.

5. The method of claim 1, wherein the modifying of the background includes changing one or more background colors.

6. The method of claim 1, wherein the modifying of the background includes changing a background resolution.

7. The method of claim 1, wherein the modifying of the background includes changing a video dot density.

8. The method of claim 1, wherein the modifying of the background includes changing posterization or pixelization of the background.

9. The method of claim 1, wherein the modifying of the background includes replacing the background with a predetermined image.

10. The method of claim 1, wherein the at least one object of interest includes at least a portion of an individual other than a human face.

11. The method of claim 1, wherein the at least one object of interest includes a human face.

12. The method of claim 1, further comprising transmitting the modified video over a communications network.

13. The method of claim 1, further comprising receiving a request to blur the background of the video.

14. The method of claim 1, further comprising:

monitoring quality of service associated with a communications network; and

based on the monitoring, generating a request to blur the background of the video.

15. The method of claim 1, further comprising:

dynamically monitoring a network parameter associated with transferring of the video over a communications network;

if the network parameter is below a predetermined threshold value, generating a request to blur the background of the video; and

if the network parameter is above the predetermined threshold value, generating a request to transmit the video without blurring.

16. The method of claim 15, wherein the network parameter includes a bit rate or a network bandwidth.

17. The method of claim 15, wherein the modifying of the background includes gradual blurring of the background, wherein a degree of the gradual blurring depends on the network parameter.

18. The method of claim 1, further comprising:

dynamically determining a value related to quality of service associated with a communications network;

based on the determining, if the value associated with the quality of service is within a first predetermined range, generating a first request to blur only the background of the video;

based on the determining, if the value associated with the quality of service is within a second predetermined range, generating a second request to blur the background of the video and other parts of the video excluding a user face; and

based on the determining, if the value associated with the quality of service is within a third predetermined range, no request to blur the background is generated; and

wherein the first range differs from the second range and the third range, and wherein the second range differs from the third range and the first range.

19. The method of claim 1, wherein the identifying of the at least one object of interest includes applying a Viola-Jones algorithm to the images.

20. The method of claim 1, wherein the detecting of the feature reference points includes applying an Active Shape Model algorithm to areas of the images associated with the at least one object of interest.

21. The method of claim 1, wherein the identifying of the background comprises:

forming a binary mask associated with the at least one object of interest;

aligning the binary mask to a virtual face mesh on each image; and

creating an inverted binary mask by inverting the binary mask.

22. The method of claim 21, wherein the forming of the binary mask comprises:

determining a gray value intensity of a plurality of image sections in each of the images, wherein the plurality of image sections are associated with the virtual face mesh;

determining object pixels associated with the object of interest by comparing the gray value intensity of each of the image sections with a reference value;

applying a binary morphological closing algorithm to the object pixels; and

removing unwanted pixel conglomerates from the virtual face mesh.

23. The method of claim 21, wherein the aligning of the binary mask to the virtual face mesh comprises:

making a projection of the virtual face mesh to a reference grid thereby separating the virtual face mesh into a plurality of reference grid cells;

associating virtual face mesh elements which correspond to reference grid cells; and

determining pixels of each of the images which correspond to the virtual face mesh elements.

24. The method of claim 1, further comprising modifying image portions associated with the at least one object of interest in each of the images.

25. The method of claim 24, wherein the modifying of the image portions associated with the at least one object of interest is based on the feature reference points;

and

wherein the modifying of the image portions associated with the at least one object of interest includes changing at least one of a color, a color tone, a proportion, and a resolution.

26. The method of claim 24, wherein the modifying of the image portions associated with the at least one object of interest includes smoothing.

27. The method of claim 1, wherein the identifying of the at least one object of interest in each of the images is based on a user input.

28. The method of claim 1, further comprising:

determining a position of a head based on the identifying of the at least one object of interest and the reference feature points;

determining a position of a body based on the position of the head; and

tracking the position of the body over the sequence of images.

29. A system, comprising:

a computing device including at least one processor and a memory storing processor-executable codes, which, when implemented by the at least one processor, cause to perform the steps of;

receiving a video including a sequence of images;

identifying at least one object of interest in one or more of the images;

detecting at least one shape unit and a position vector of the at least one object of interest, and an intensity vector of the at least one shape unit, the at least one shape unit representing a parameter of a face of the at least one object of interest and the position vector corresponding to a rotation around three axes and a translation along the axes;

tracking the at least one object of interest in the video, wherein the tracking is based on the at least one shape unit, the position vector, and the intensity vector;

identifying a background in each of the images by separating the at least one object of interest from each image based on the at least one shape unit, the position vector, and the intensity vector;

modifying the background in each of the images, thereby generating a modified background; and

generating a modified video which includes the at least one object of interest and the modified background.

30. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method, comprising:

receiving a video including a sequence of images;

identifying at least one object of interest in one or more of the images;

detecting at least one shape unit and a position vector of the at least one object of interest, and an intensity vector of the at least one shape unit, the at least one shape unit representing a parameter of a face of the at least one object of interest and the position vector corresponding to a rotation around three axes and a translation along the axes;

tracking the at least one object of interest in the video, wherein the tracking is based on the at least one shape unit, the position vector, and the intensity vector;

identifying a background in each of the images by separating the at least one object of interest from each image based on the at least one shape unit, the position vector, and the intensity vector;

modifying the background in each of the images, thereby generating a modified background; and

generating a modified video which includes the at least one object of interest and the modified background.