Facial tracking with classifiers

• US 10,614,289 B2
• Filed: 09/08/2015
• Issued: 04/07/2020
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: Senechal, Thibaud, el Kaliouby, Rana, Turcot, Panu James
• Primary Examiner(s): Lee, Jonathan S
• Application Number:
  US14/848,222
• Publication Number:
  US 20160004904A1
• Time in Patent Office: 1,673 Days
• Field of Search: 382103
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/50 for simulation or modelling of medical disorders
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Concepts for facial tracking with classifiers is disclosed. One or more faces are detected and tracked in a series of video frames that include at least one face. Video is captured and partitioned into the series of frames. A first video frame is analyzed using classifiers trained to detect the presence of at least one face in the frame. The classifiers are used to initialize locations for a first set of facial landmarks for the first face. The locations of the facial landmarks are refined using localized information around the landmarks, and a rough bounding box that contains the facial landmarks is estimated. The future locations for the facial landmarks detected in the first video frame are estimated for a future video frame. The detection of the facial landmarks and estimation of future locations of the landmarks are insensitive to rotation, orientation, scaling, or mirroring of the face.

Claims

1. A computer-implemented method for facial detection comprising:

obtaining a video that includes a face;

performing face detection to initialize locations for a first set of facial landmarks within a first frame from the video containing an image of the face wherein the face detection comprises;

performing facial landmark detection within the first frame from the video; and

estimating a rough bounding box for the face based on the facial landmark detection;

refining the locations for the first set of facial landmarks based on localized information around the first set of facial landmarks;

estimating future locations for landmarks within the first set of facial landmarks for a future frame from the first frame; and

training a classifier for a video clip for facial detection, wherein the training includes generating a scaled version of the image of the face directly from an initial frame, wherein the video is a chronological series of frames and the initial frame is an earliest frame containing the face in the chronological series of frames.

2. The method of claim 1 wherein the estimating of the future locations for the landmarks is based on a velocity for one or more of the locations.

3. The method of claim 1 wherein the estimating of the future locations for the landmarks is based on an angular velocity for one or more of the locations.

4. The method of claim 1 further comprising providing an output for a facial detector based on the estimating of the future locations for the landmarks.

5. The method of claim 1 further comprising performing face detection to initialize a second set of locations for a second set of facial landmarks for a second face within the video.

6. The method of claim 5 wherein the performing face detection on the second face comprises:

performing facial landmark detection within the first frame from the video for the second face; and

estimating a second rough bounding box for the second face based on the facial landmark detection.

7. The method of claim 6 further comprising refining the second set of locations for the second set of facial landmarks based on localized information around the second set of facial landmarks.

8. The method of claim 7 further comprising estimating future locations for the second set of locations for the second set of facial landmarks for the future frame from the first frame.

9. The method of claim 6 further comprising distinguishing facial points from the first face from other facial points.

10. The method of claim 9 wherein the other facial points correspond to the second face.

11. The method of claim 9 wherein one or more of the other facial points correspond to a third face.

12. The method of claim 1 further comprising analyzing the face using a plurality of classifiers.

13. The method of claim 12 wherein the plurality of classifiers provides for analysis of gender, ethnicity, or age corresponding to the face.

14. The method of claim 1 further comprising generating a bounding box for the face within the first frame.

15. The method of claim 1 wherein the training includes generating a mirror image of the face.

16. The method of claim 1 wherein the training includes generating a rotated image of the face.

17. The method of claim 1 wherein the training includes translating the bounding box to a different location.

18. The method of claim 1 further comprising evaluating the face to determine rotation about a z-axis of the face.

19. The method of claim 1 further comprising estimating a quality of the bounding box for the future frame.

22. The method of claim 1 wherein generating a scaled version of the image of the face comprises generating a zoomed-in or enlarged version of the image.

23. The method of claim 1 wherein generating a scaled version of the image of the face comprises generating a zoomed-out or shrunken version of the image.

24. The method of claim 7 wherein the refining the second set of locations for the second set of facial landmarks includes centering location points on the second set of facial landmarks.

25. The method of claim 1 wherein the future frame is a subsequent frame in the chronological series of frames from the first frame.

26. The method of claim 1 wherein the first frame from the video is the initial frame.

20. A computer program product embodied in a non-transitory computer readable medium for facial detection comprising:

code for obtaining a video that includes a face;

code for performing face detection to initialize locations for a first set of facial landmarks within a first frame from the video containing an image of the face wherein the face detection comprises;

performing facial landmark detection within the first frame from the video; and

estimating a rough bounding box for the face based on the facial landmark detection;

code for refining the locations for the first set of facial landmarks based on localized information around the first set of facial landmarks;

code for estimating future locations for landmarks within the first set of facial landmarks for a future frame from the first frame; and

code for training a classifier for a video clip for facial detection, wherein the training includes generating a scaled version of the image of the face directly from an initial frame, wherein the video is a chronological series of frames and the initial frame is an earliest frame containing the face in the chronological series of frames.

21. A computer system for facial detection comprising:

a memory which stores instructions;

one or more processors attached to the memory wherein the one or more processors when executing the instructions which are stored, are configured to;

obtain a video that includes a face;

perform face detection to initialize locations for a first set of facial landmarks within a first frame from the video containing an image of the face wherein the face detection comprises;

performing facial landmark detection within the first frame from the video; and

estimating a rough bounding box for the face based on the facial landmark detection;

refine the locations for the first set of facial landmarks based on localized information around the first set of facial landmarks;

estimate future locations for landmarks within the first set of facial landmarks for a future frame from the first frame; and

train a classifier for a video clip for facial detection, wherein the training includes generating a scaled version of the image of the face directly from an initial frame, wherein the video is a chronological series of frames and the initial frame is an earliest frame containing the face in the chronological series of frames.