Face detection for background management

• US 10,122,965 B2
• Filed: 11/29/2016
• Issued: 11/06/2018
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Taine, Stephane, Aronoff, Brendan Benjamin, Clark, Jason
• Primary Examiner(s): ZENATI, AMAL S
• Application Number:
  US15/364,191
• Publication Number:
  US 20180152667A1
• Time in Patent Office: 707 Days
• Field of Search: 348 1407
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Exemplary embodiments relate to uses of face detection for modifying the background of a video. By detecting the portion of the video shot that corresponds to a face, it can also determined which portion of the shot do not correspond to the face (e.g., the background of the video). The thus-detected background may be replaced with a movie, event, or other material. Different types of materials may be targeted at different users based on user affinities or preferences. In some embodiments, a user may manually select and apply a background during a video call. When applying a custom background, the system can transmit a control signal and the background. The client recognizes the control signal and, in response, applies the background.

Claims

1. A method, comprising:

accessing one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;

performing face detection to detect a portion of the one or more frames that substantially comprises the face;

based on the face detection, assigning an area of the one or more frames not belonging to the face as a background of the one or more frames;

accessing a video channel to a server associated with the video call, accessing an audio channel to the server associated with the video call, and accessing a control channel to the server, the control channel for carrying control data and distinct from the video channel and the audio channel;

forwarding the image data to the server on the video channel and forwarding the audio data to the server on the audio channel; and

transmitting a control signal to the server on the control channel distinct from the video channel and the audio channel, the control signal comprising an instruction to modify the background, an identification of the one or more frames subject to modification, and at least one of a copy of the modified background or a background identifier to a server associated with the video call, the transmitting occurring in a control channel distinct from a channel for transmitting audio and distinct from a channel for transmitting video.

2. The method of claim 1, wherein modifying the background comprises replacing the background with a new background.

3. The method of claim 1, wherein the modified background comprises material selected based on one or more characteristics associated with a user whose face appears in the one or more frames.

4. The method of claim 1, wherein the modified background includes variable promotional material.

5. The method of claim 1, wherein modifying the background is performed in response to receiving a background modification instruction in an interface of the video call.

6. The method of claim 1, further comprising:

receiving a video signal comprising video from a sender client, a control signal associated with the video signal, and a copy of a background or a background identifier associated with the control signal; and

in response to receiving the control signal, performing sender face detection on the video signal to identify a sender face in the video signal;

based on the sender face detection, assigning an area of the one or more frames not belonging to the sender face as a sender background of the video signal; and

replacing the sender background with the copy of the background or with a local copy of a background associated with the background identifier.

7. A non-transitory computer-readable medium storing instructions configured to cause one or more processors to:

access one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;

perform face detection to detect a portion of the one or more frames that substantially comprises the face;

based on the face detection, assign an area of the one or more frames not belonging to the face as a background of the one or more frames;

access a video channel to a server associated with the video call, accessing an audio channel to the server associated with the video call, and accessing a control channel to the server, the control channel for carrying control data and distinct from the video channel and the audio channel;

forward the image data to the server on the video channel and forwarding the audio data to the server on the audio channel; and

transmit a control signal to the server on the control channel distinct from the video channel and the audio channel, the control signal comprising an instruction to modify the background, an identification of the one or more frames subject to modification, and at least one of a copy of the modified background or a background identifier to a server associated with the video call, the transmitting occurring in a control channel distinct from a channel for transmitting audio and distinct from a channel for transmitting video.

8. The medium of claim 7, wherein modifying the background comprises replacing the background with a new background.

9. The medium of claim 7, wherein the modified background comprises material selected based on one or more characteristics associated with a user whose face appears in the one or more frames.

10. The medium of claim 7, wherein the modified background includes variable promotional material.

11. The medium of claim 7, wherein modifying the background is performed in response to receiving a background modification instruction in an interface of the video call.

12. The medium of claim 7, further storing instructions for:

receiving a video signal comprising video from a sender client, a control signal associated with the video signal, and a copy of a background or a background identifier associated with the control signal; and

in response to receiving the control signal, performing sender face detection on the video signal to identify a sender face in the video signal;

based on the sender face detection, assigning an area of the one or more frames not belonging to the sender face as a sender background of the video signal; and

replacing the sender background with the copy of the background or with a local copy of a background associated with the background identifier.

13. An apparatus comprising:

a non-transitory computer readable medium configured to store instructions for facilitating a video call; and

a processor configured to execute the instructions, the instructions configured to cause the processor to;

access one or more frames from a video call, the video call comprising image data and audio data, the one or more frames comprising a face;

perform face detection to detect a portion of the one or more frames that substantially comprises the face;

based on the face detection, assign an area of the one or more frames not belonging to the face as a background of the one or more frames;

access a video channel to a server associated with the video call, accessing an audio channel to the server associated with the video call, and accessing a control channel to the server, the control channel for carrying control data and distinct from the video channel and the audio channel;

forward the image data to the server on the video channel and forwarding the audio data to the server on the audio channel; and

transmit a control signal to the server on the control channel distinct from the video channel and the audio channel, the control signal comprising an instruction to modify the background, an identification of the one or more frames subject to modification, and at least one of a copy of the modified background or a background identifier to a server associated with the video call, the transmitting occurring in a control channel distinct from a channel for transmitting audio and distinct from a channel for transmitting video.

14. The apparatus of claim 13, wherein modifying the background comprises replacing the background with a new background.

15. The apparatus of claim 13, wherein the modified background comprises material selected based on one or more characteristics associated with a user whose face appears in the one or more frames.

16. The apparatus of claim 13, wherein the modified background includes variable promotional material.

17. The apparatus of claim 13, wherein modifying the background is performed in response to receiving a background modification instruction in an interface of the video call.