Head-pose invariant recognition of facial attributes

• US 9,852,327 B2
• Filed: 01/13/2017
• Issued: 12/26/2017
• Est. Priority Date: 07/17/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Emotient, Inc. (Apple Inc.)
• Inventors: Whitehill, Jacob, Movellan, Javier R., Fasel, Ian
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US15/406,034
• Publication Number:
  US 20170213075A1
• Time in Patent Office: 347 Days
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A system facilitates automatic recognition of facial expressions or other facial attributes. The system includes a data access module and an expression engine. The expression engine further includes a set of specialized expression engines, a pose detection module, and a combiner module. The data access module accesses a facial image of a head. The set of specialized expression engines generates a set of specialized expression metrics, where each specialized expression metric is an indication of a facial expression of the facial image assuming a specific orientation of the head. The pose detection module determines the orientation of the head from the facial image. Based on the determined orientation of the head and the assumed orientations of each of the specialized expression metrics, the combiner module combines the set of specialized expression metrics to determine a facial expression metric for the facial image that is substantially invariant to the head orientation.

Claims

1. A computer-implemented method for training a combination engine for facial recognition, the method comprising:

receiving a first training facial image of a head from a plurality of training facial images, the first training facial image having a preselected attribute within a first range of head poses associated with a set of specialized recognition engines (SREs);

receiving a first set of specialized recognition metrics (SRMs) associated with the first training facial image, each SRM of the first set of SRMs associated with a SRE of the set of SREs;

receiving first head pose information for the first training facial image;

determining a first facial expression metric based on the first set of SRMs and the received first head pose information;

associating the first facial expression metric with the received first head pose information.

2. The computer implemented method of claim 1, wherein the received first head pose information includes an indication that a head pose of the first training facial image is within a range associated with a SRE of the set of SREs.

3. The computer-implemented method of claim 1, wherein each SRM of the first set of SRMs is an indication of a facial expression of the first facial image assuming a specific orientation of the head, and each SRM corresponds to different orientations of the head.

4. The computer-implemented method of claim 1, wherein the first range of head poses associated with each SRM spans within ten degrees with respect to at least one of Euler angles (yaw, pitch, roll).

5. The computer implemented method of claim 1, further comprising:

receiving a second training facial image from the plurality of training facial images, the second training facial image having the preselected attribute, and having a second range of head poses associated with the set of SREs, the second range of head poses different from the first range of head poses;

receiving second head pose information; and

determining a second facial expression metric based on a second set of SRMs and the received second head pose information, the second facial expression metric equal to the first facial expression metric.

6. The computer-implemented method of claim 5, wherein the head pose for the second facial image differs by at least five degrees from the head pose for the first facial image.

7. The computer-implemented method of claim 5, further comprising determining a set of weights for the second set of SRMs, wherein the determined set of weights is based on the first facial expression metric.

8. The computer-implemented method of claim 1, wherein the determining the first facial expression metric is based on weights assigned to each SRM of the first set of SRMs based on the received first head pose information.

9. The computer-implemented method of claim 8, wherein the weights are assigned based, at least in part, on how closely received first head pose information matches an orientation corresponding to the first range of head poses associated with each SRM of the first set of SRMs.

10. The computer-implemented method of claim 9, wherein the facial metric is based on the weighted sum of each SRM of the first set of SRMs.

11. A non-transitory computer readable medium containing instructions that, when executed by a processor, cause a programmable device to:

receive a first training facial image of a head from a plurality of training facial images, the first training facial image having a preselected attribute within a first range of head poses associated with a set of specialized recognition engines (SREs);

receive a first set of specialized recognition metrics (SRMs) associated with the first training facial image, each SRM of the first set of SRMs associated with a SRE of the set of SREs;

receive first head pose information for the first training facial image;

determine a first facial expression metric based on the first set of SRMs and the received first head pose information;

associate the first facial expression metric with the received first head pose information.

12. The computer readable medium of claim 11, wherein the received first head pose information includes an indication that a head pose of the first training facial image is within a range associated with a SRE of the set of SREs.

13. The computer readable medium of claim 11, wherein each SRM of the first set of SRMs is an indication of a facial expression of the first facial image assuming a specific orientation of the head, and each SRM corresponds to different orientations of the head.

14. The computer readable medium of claim 11, wherein the first range of head poses associated with each SRM spans within ten degrees with respect to at least one of Euler angles (yaw, pitch, roll).

15. The computer readable medium of claim 11, wherein the instructions further comprise instructions that when executed cause the programmable device to:

receive a second training facial image from the plurality of training facial images, the second training facial image having the preselected attribute, and having a second range of head poses associated with the set of SREs, the second range of head poses different from the first range of head poses;

receive second head pose information; and

determine a second facial expression metric based on a second set of SRMs and the received second head pose information, the second facial expression metric equal to the first facial expression metric.

16. The computer readable medium of claim 15, wherein the head pose for the second facial image differs by at least five degrees from the head pose for the first facial image.

17. The computer readable medium of claim 15, wherein the instructions further comprise instructions that when executed cause the programmable device to determine a set of weights for the second set of SRMs, wherein the determined set of weights is based on the first facial expression metric.

18. The computer readable medium of claim 11, wherein the instructions that cause a programmable device to determine the first facial expression metric comprise instructions that cause the programmable device to determine the first facial expression metric based on weights assigned to each SRM of the first set of SRMs based on the received first head pose information.

19. The computer readable medium of claim 18, wherein the weights are assigned based, at least in part, on how closely received first head pose information matches an orientation corresponding to the first range of head poses associated with each SRM of the first set of SRMs.

20. A computing device comprising:

one or more processors; and

a memory, operatively coupled to the one or more processors, wherein the memory is configured to store instructions that, when executed by the one or more processors, cause the one or more processors to;

receive a first training facial image of a head from a plurality of training facial images, the first training facial image having a preselected attribute within a first range of head poses associated with a set of specialized recognition engines (SREs);

receive a first set of specialized recognition metrics (SRMs) associated with the first training facial image, each SRM of the first set of SRMs associated with a SRE of the set of SREs;

receive first head pose information for the first training facial image;

determine a first facial expression metric based on the first set of SRMs and the received first head pose information;

associate the first facial expression metric with the received first head pose information.