Face detection for video calls

• US 10,303,928 B2
• Filed: 11/29/2016
• Issued: 05/28/2019
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Taine, Stephane, Aronoff, Brendan Benjamin, Clark, Jason
• Primary Examiner(s): Zenati, Amal S
• Application Number:
  US15/364,188
• Publication Number:
  US 20180152666A1
• Time in Patent Office: 910 Days
• Field of Search: 348 1407
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Exemplary embodiments relate to uses of face detection in video, and especially in video calls. In some embodiments, face detection may be used to center a camera shot by maintaining a face in the center of a screen. The centering may be applied selectively, such as by overriding centering if the user is looking off-screen. The video may also be cropped to better fit a face in a screen, or to allow multiple faces to appear on screen. In some embodiments, emphasizing the face over the background (or parts of the face over the whole face) allows for improvement in video call performance. Moreover, these techniques can be used to bring certain areas of a camera shot into focus while de-emphasizing the background (or vice versa).

Claims

1. A method, comprising:

accessing one or more frames from a video call, the one or more frames comprising a frame view containing a face;

performing face detection to detect dimensions and a position of the face in the one or more frames, the video call configured to perform a cropping of the frame view based on the dimensions detected in the face detection and a centering of at least the face in the frame view based on the position detected in the face detection;

determining that the face has shifted such as to be oriented to look at an area outside of the frame view and that the frame view has moved to include both the face and the area to which the face is oriented to look; and

refraining from performing at least one of the cropping or the centering in subsequent frames based on the determination that the face is positioned to look at the area outside of the frame view and that the frame view has moved.

2. The method of claim 1, the face occupying substantially a predetermined amount of a display upon cropping, and the face occupying substantially a center portion of the display upon centering.

3. The method of claim 1, wherein the face is one of a plurality of faces in the frame, the cropping causing the plurality of faces to occupy substantially a predetermined amount of a display and the centering causing the plurality of faces to be collectively centered in the display.

4. The method of claim 1, further comprising:

performing feature detection on the face, the feature detection identifying a first portion of the face comprising at least one of an eye or a mouth and a second portion of the face not comprising an eye or a mouth; and

emphasizing the first portion of the face over the second portion of the face, the emphasizing comprising at least one of blurring the second portion, rendering the second portion at a lower resolution than the first portion, or quantizing the second portion differently than the first portion.

5. The method of claim 1, wherein the video call comprises a key frame and a current frame, and further comprising:

determining differences between the key frame and the current frame, the differences comprising a first difference relating to a portion of the current frame that is a part of the face and a second difference relating to a portion of the current frame that is a part of the background;

quantizing the first difference and the second difference, the first difference being more highly quantized than the second difference; and

generating a delta frame that includes the quantized differences.

6. A non-transitory computer-readable medium storing instructions configured to cause one or more processors to:

access one or more frames from a video call, the one or more frames comprising a frame view containing a face;

perform face detection to detect dimensions and a position of the face in the one or more frames;

perform a cropping of the frame view based on the dimensions detected in the face detection;

perform a centering of at least the face in the frame view based on the position detected in the face detection;

determine that the face has shifted such as to be oriented to look at an area outside of the frame view and that the frame view has moved to include both the face and the area to which the face is oriented to look;

refrain from performing at least one of the cropping or the centering in subsequent frames based on the determination that the face is positioned to look at the area outside of the frame view and that the frame view has moved.

7. The medium of claim 6, the face occupying substantially a predetermined amount of a display upon cropping, and the face occupying substantially a center portion of the display upon centering.

8. The medium of claim 6, wherein the face is one of a plurality of faces in the frame, the cropping causing the plurality of faces to occupy substantially a predetermined amount of a display and the centering causing the plurality of faces to be collectively centered in the display.

9. The medium of claim 6, further storing instructions for:

performing feature detection on the face, the feature detection identifying a first portion of the face comprising at least one of an eye or a mouth and a second portion of the face not comprising an eye or a mouth; and

emphasizing the first portion of the face over the second portion of the face, the emphasizing comprising at least one of blurring the second portion, rendering the second portion at a lower resolution than the first portion, or quantizing the second portion differently than the first portion.

10. The medium of claim 6, wherein the video call comprises a key frame and a current frame, and further storing instructions for:

determining differences between the key frame and the current frame, the differences comprising a first difference relating to a portion of the current frame that is a part of the face and a second difference relating to a portion of the current frame that is a part of the background;

quantizing the first difference and the second difference, the first difference being more highly quantized than the second difference; and

generating a delta frame that includes the quantized differences.

11. An apparatus comprising:

a non-transitory computer readable medium configured to store instructions for facilitating a video conversation; and

a processor configured to execute the instructions, the instructions configured to cause the processor to;

access one or more frames from the video conversation, the one or more frames comprising a frame view containing a face;

perform face detection to detect dimensions and a position of the face in the frame view;

perform a cropping of the frame view based on the dimensions detected in the face detection;

perform a centering of at least the face in the frame view based on the position detected in the face detection;

determine that the face has shifted such as to be oriented to look at an area outside of the frame view and that the frame view has moved to include both the face and the area to which the face is oriented to look; and

refrain from performing at least one of the cropping or the centering in subsequent frames based on the determination that the face is positioned to look at the area outside of the frame view and that the frame view has moved.

12. The apparatus of claim 11, wherein the face is one of a plurality of faces in the frame, the cropping causing the plurality of faces to occupy substantially a predetermined amount of a display and the centering causing the plurality of faces to be collectively centered in the display.

13. The apparatus of claim 11, the instructions further configured to cause the processor to:

performing feature detection on the face, the feature detection identifying a first portion of the face comprising at least one of an eye or a mouth and a second portion of the face not comprising an eye or a mouth; and

emphasizing the first portion of the face over the second portion of the face, the emphasizing comprising at least one of blurring the second portion, rendering the second portion at a lower resolution than the first portion, or quantizing the second portion differently than the first portion.

14. The apparatus of claim 11, wherein the video conversation comprises a key frame and a current frame, and the instructions further configured to cause the processor to:

determining differences between the key frame and the current frame, the differences comprising a first difference relating to a portion of the current frame that is a part of the face and a second difference relating to a portion of the current frame that is a part of the background;

quantizing the first difference and the second difference, the first difference being more highly quantized than the second difference; and

generating a delta frame that includes the quantized differences.