Method and system for automatically recognizing facial expressions via algorithmic periocular localization

• US 9,996,737 B2
• Filed: 03/16/2017
• Issued: 06/12/2018
• Est. Priority Date: 08/29/2012
• Status: Active Grant

• Current Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Original Assignee: Conduent Business Services, LLC (Conduent, Inc.)
• Inventors: Shreve, Matthew Adam, Mongeon, Michael C., Loce, Robert P., Bernal, Edgar A., Wu, Wencheng
• Primary Examiner(s): Lee, Michael
• Assistant Examiner(s): Satti, Humam
• Application Number:
  US15/460,848
• Publication Number:
  US 20170185826A1
• Time in Patent Office: 453 Days
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression

Abstract

This disclosure provides a method and system for automatically recognizing facial expressions at variable resolutions of video. According to one exemplary method, facial expressions are detected, extracted and classified from a video sequence based on an automatic localization of the periocular region associated with a detected and extracted face.

Claims

1. An image capturing system comprising:

an image capturing device; and

an image processor operatively associated with the image capturing device, the image processor configured to perform a method of detecting, extracting and classifying a facial expression associated with a human face, the method comprising;

a) acquiring a surveillance video sequence using at least one video camera operatively connected to a computer, the video sequence including one or more video frames including a human face;

b) detecting in the video sequence the one or more video frames including the human face and extracting a facial region associated with the detected human face;

c) automatically detecting and localizing within the extracted facial region a periocular region associated with the extracted facial region by initially estimating the location of the periocular region within the extracted facial region based on an average geometric relationship of an average periocular region relative to an average extracted facial region associated with an average human face, and subsequently processing a subregion of the extracted facial region substantially centered about the initial estimate of the location of the periocular region to calculate an actual location of the periocular region associated with the extracted facial region and segmenting the extracted facial region of the video frame including the human face about the actual location of the detected and localized periocular region based on an average geometric relationship of the average human face relative to a location of the average periocular region and the segmented facial region including a greater total number of pixels than the periocular region;

d) processing the segmented facial region associated with the video frame including the human face to a predetermined lighting and scale condition;

e) extracting representative features associated with the segmented facial region; and

f) classifying automatically the expression of the segmented facial region by querying the extracted representative features associated with the segmented facial region into an expression classifier, the expression classifier outputting one of the plurality of expression classes associated with the segmented facial region.

2. The image capturing system according to claim 1, wherein the image capturing system is operatively associated with a retail environment and the human face is associated with one or both of a customer and an employee.

3. The image capturing system according to claim 1, wherein the image capturing system is operatively associated with a security surveillance system.

4. The image capturing system according to claim 1, step f) comprising:

accessing a support vector machine associated with a facial expression database, and classifying the appearance of the human face associated with the video frame based on inputting an extracted feature vector to the support vector machine.

5. The image capturing system according to claim 1, wherein a resolution of the acquired video sequence is substantially 640×

480 or lower.

6. The image capturing system according to claim 1, further comprising:

repeating steps a)-f) for a plurality of temporal spaced video frames including the human face; and

detecting if the facial expression associated with the human face changed over the span of the temporal spaced video frames.

7. The image capturing system according to claim 1, wherein step b) uses a Haar-feature classifier trained to detect human faces.

8. The image capturing system according to claim 1, wherein step e) uses one of LBP (local binary pattern), HOG (histogram of gradients), SIFT (scale invariant feature transform), and SURF (speeded up robust features) calculations to extract a feature vector from the second pixel region.