Feature descriptor for robust facial expression recognition

• US 9,239,948 B2
• Filed: 07/10/2013
• Issued: 01/19/2016
• Est. Priority Date: 07/17/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Sudha, Velusamy, Balasubramanian, Anand, Viswanath, Gopalakrishnan, Pratibha, Moogi
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US13/938,371
• Publication Number:
  US 20140023269A1
• Time in Patent Office: 923 Days
• Field of Search: 382/118, 382/170
• US Class Current: 1/1
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A method and system for generating a feature descriptor for robust facial expression recognition pre-processes a facial image using a Gaussian filter to smooth the facial image. Then, gradient based images at M scales and N orientations are generated from the pre-processed facial image. Further, a portion of an image corresponding to each action unit is selected from each of the gradient based images. Thereafter, appearance of at least one facial event in the selected portion of the image is captured. Also, a geometry of the at least one facial event in the selected portion of image is determined to obtain a feature descriptor for each action unit for robust facial expression recognition.

Claims

1. A method of generating a feature descriptor for facial expression recognition, the method comprising:

generating a plurality of gradient based images with different scales and orientations from a facial image;

selecting a portion of an image corresponding to each action unit in each of the plurality of gradient based images;

determining, by a processor, an appearance of at least one facial event in the selected portion of the image using coded appearance values; and

determining a geometry of the at least one facial event in the selected portion of the image using a spin wheel structure to obtain a feature descriptor for each action unit for facial expression recognition.

2. The method of claim 1, wherein determining the appearance of the at least one facial event in the selected portion of the image comprises:

computing a coded appearance value for each pixel in the selected portion of the image based on relative intensity of each pixel with respect to neighboring pixels, wherein the code assigned to each pixel is indicative of the appearance of the at least one facial event.

3. The method of claim 2, wherein determining the geometry of the at least one facial event in the selected portion comprises:

dividing the selected portion into a pre-defined number of blocks of different shapes;

constructing a spin wheel structure in each of the blocks, wherein the spin wheel structure comprises a plurality of segments, and wherein each of the segments comprises a set of pixels belonging to each respective block;

generating a histogram for each of the segments in each spin wheel structure in the selected portion of the image based on the codes assigned to the set of pixels in each segment; and

generating a concatenated histogram for the portion of the image from the histograms associated with the segments in each spin wheel structure in the selected portion of the image.

4. The method of claim 1, further comprising:

pre-processing the facial image using a Gaussian filter.

5. The method of claim 1, wherein computing the code for each pixel in the selected portion of the image comprises:

computing a code for each pixel in the selected portion of the image by integrating relative information around each pixel.

6. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises a facial descriptor generation module, wherein the facial descriptor generation module instructs the processor to perform a method comprising;

generating a plurality of gradient based images with different scales and orientations from a facial image;

selecting a portion of an image corresponding to each action unit in each of the plurality of gradient based images;

determining an appearance of at least one facial event in the selected portion of the image using coded appearance values; and

determining a geometry of the at least one facial event in the selected portion of the image using a spin wheel structure to obtain a feature descriptor for each action unit for facial expression recognition.

7. The apparatus of claim 6, wherein the method further comprises:

computing a coded appearance value for each pixel in the selected portion of the image based on relative intensity of each pixel with respect to neighboring pixels, wherein the code assigned to each pixel is indicative of the appearance of the at least one facial event.

8. The apparatus of claim 7, wherein the method further comprises:

dividing the selected portion into a pre-defined number of blocks of different shapes;

constructing a spin wheel structure in each of the blocks, wherein the spin wheel structure comprises a plurality of segments, and wherein each of the segments comprises a set of pixels belonging to each respective block;

generating a histogram for each of the segments in each spin wheel structure in the selected portion of the image based on the codes assigned to the set of pixels in each segment; and

generating a concatenated histogram for the portion of the image from the histograms associated with the segments in each spin wheel structure in the selected portion of the image.

9. The apparatus of claim 6, wherein the method further comprises:

pre-processing the facial image using a Gaussian filter.

10. The apparatus of claim 6, wherein the method further comprises:

computing a code for each pixel in the selected portion of the image by integrating relative information around each pixel.

11. A non-transitory computer-readable storage medium having executable program stored therein, that when executed by a processor of a computing device, cause the computing device to perform the method comprising:

generating a plurality of gradient based images with different scales and orientations from a facial image;

selecting a portion of an image corresponding to each action unit in each of the plurality of gradient based images;

determining an appearance of at least one facial event in the selected portion of the image using coded appearance values; and

determining a geometry of the at least one facial event in the selected portion of the image using a spin wheel structure to obtain a feature descriptor for each action unit for facial expression recognition.

12. The storage medium of claim 11, wherein the method further comprises:

computing a coded appearance value for each pixel in the selected portion of the image based on relative intensity of each pixel with respect to neighboring pixels, wherein the code assigned to each pixel is indicative of the appearance of the at least one facial event.

13. The storage medium of claim 12, wherein the method further comprises:

dividing the selected portion into a pre-defined number of blocks of different shapes;

constructing a spin wheel structure in each of the blocks, wherein the spin wheel structure comprises a plurality of segments, and wherein each of the segments comprises a set of pixels belonging to each respective block;

generating a histogram for each of the segments in each spin wheel structure in the selected portion of the image based on the codes assigned to the set of pixels in each segment; and

generating a concatenated histogram for the portion of the image from the histograms associated with the segments in each spin wheel structure in the selected portion of the image.

14. The storage medium of claim 11, wherein the method further comprises:

pre-processing the facial image using a Gaussian filter.

15. The storage medium of claim 11, wherein the method further comprises:

computing a code for each pixel in the selected portion of the image by integrating relative information around each pixel.

16. A method of recognizing a facial expression of an image, the method comprising:

generating a plurality of gradient-based images from the image;

selecting a portion, corresponding to an action unit, from one of the plurality of gradient-based images;

determining, by a processor, an appearance of a facial event in the selected portion of the image using coded appearance values; and

determining a geometry of the facial event using a spin wheel structure to obtain a feature descriptor for the action unit.

17. A non-transitory computer-readable recording medium storing a program to implement the method of claim 16.