AU FEATURE RECOGNITION METHOD AND DEVICE, AND STORAGE MEDIUM

• US 20190228211A1
• Filed: 09/30/2017
• Published: 07/25/2019
• Est. Priority Date: 08/17/2017
• Status: Active Grant

• Current Assignee: Ping An Technology Company Limited (Ping An Insurance (Group) Company of China, Ltd.)
• Original Assignee: Ping An Technology Company Limited (Ping An Insurance (Group) Company of China, Ltd.)
• Inventors: Chen, Lin, Zhang, Guohui
• Granted Patent: US 10,445,562 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

The disclosure discloses an action unit (AU) feature recognition method, which includes: acquiring a real-time image shot by a photographic device and extracting a real-time facial image from the real-time image by use of a face recognition algorithm; inputting the real-time facial image into a pretrained facial mean shape and recognizing t facial feature points from the real-time facial image by use of the facial mean shape; determining feature regions matched with each AU in the real-time facial image according to positions of the t facial feature points, extracting local features from the feature regions and generating multiple feature vectors; and inputting the multiple feature vectors into pretrained AU classifiers matched with the feature regions respectively to obtain a prediction result of recognition of the corresponding AU features from the feature regions. The disclosure also discloses an electronic device and a computer-readable storage medium.

Claims

1. An electronic device, comprising:

a memory, a processor and a photographic device, an action unit (AU) feature recognition program being stored in the memory and the AU feature recognition program being executed by the processor to implement the following steps of;

a real-time image capturing step;

acquiring a real-time image shot by the photographic device and extracting a real-time facial image from the real-time image by use of a face recognition algorithm;

a facial feature point recognition step;

inputting the real-time facial image into a pretrained facial mean shape and recognizing t facial feature points from the real-time facial image by use of the facial mean shape, wherein the training steps for the facial mean shape comprising;

establishing a first sample library with n facial images and marking t facial feature points at positions of eyes, eyebrows, noses, mouths and facial outer contours in each of the facial images; and

training a facial feature recognition model by use of the t facial feature points to obtain the facial mean shape, where t represents a concatenation sequence number;

a local feature extraction step;

determining feature regions matched with each AU in the real-time facial image according to positions of the t facial feature points, extracting local features from the feature regions and generating multiple feature vectors; and

an AU feature prediction step;

inputting the multiple feature vectors into pretrained AU classifiers matched with the feature regions respectively to obtain a prediction result of recognition of the corresponding AU features from the feature regions.

3. The electronic device of claim 1, wherein the AU feature recognition program is executed by the processor to further implement the following step:

a judgment step;

judging whether probabilities of each AU feature in the prediction result are higher than a preset threshold value or not.

4. The electronic device of claim 3, wherein the judgment step further comprises:

a prompting step;

when it is judged that an AU feature of which the probability is higher than the preset threshold value exists in the prediction result, making such a prompt that the AU feature is recognized from the real-time facial image.

5. The electronic device of claim 1, wherein training steps for the predetermined AU classifier comprises:

a sample preparation step;

collecting a facial sample image, cropping an image region matched with each AU from the facial sample image as a positive sample image of the AU and preparing a negative sample image for each AU;

a local feature extraction step;

extracting local features of the positive sample image and negative sample image of each AU and generating corresponding feature vectors; and

a model training step;

performing learning training on a Support Vector Machine (SVM) by use of the local feature of the positive/negative sample image of each AU to obtain the corresponding AU classifier.

2. (canceled)

6-8. -8. (canceled)

9. An action unit (AU) feature recognition method, comprising:

a real-time image capturing step;

acquiring a real-time image shot by the photographic device and extracting a real-time facial image from the real-time image by use of a face recognition algorithm;

a facial feature point recognition step;

inputting the real-time facial image into a pretrained facial mean shape and recognizing t facial feature points from the real-time facial image by use of the facial mean shape, wherein the training steps for the facial mean shape comprising;

establishing a first sample library with n facial images and marking t facial feature points at positions of eyes, eyebrows, noses, mouths and facial outer contours in each of the facial images; and

training a facial feature recognition model by use of the t facial feature points to obtain the facial mean shape, where t represents a concatenation sequence number;

a local feature extraction step;

determining feature regions matched with each AU in the real-time facial image according to positions of the t facial feature points, extracting local features from the feature regions and generating multiple feature vectors; and

an AU feature prediction step;

inputting the multiple feature vectors into pretrained AU classifiers matched with the feature regions respectively to obtain a prediction result of recognition of the corresponding AU features from the feature regions.

10. The AU feature recognition method of claim 9, wherein the facial mean shape is obtained by training a facial feature recognition model, and the facial feature recognition model is an ensemble of regression trees (ERT) algorithm and is represented by a formula as follows:

Ŝ

^t1+1=Ŝ

^t1+τ

_t1(I,Ŝ

^t1),

where t1 represents a concatenation sequence number, represents a regressor of a present stage and Ŝ

^t1 is a shape estimate of a present model;

each regressor predicts an increment τ

_t1(I,Ŝ

^t1) according to an input present image I and Ŝ

^t1 and adds the increment to the present shape estimate to improve the present model; and

in a model training process, part of feature points of all sample pictures are extracted to train a first regression tree, a second tree is trained by use of residuals between predicted values of the first regression tree and true values of the part of feature points, and so on, when residuals of predicted values of an Nth trained tree and the true values of the part of feature points are close to zero, all regression trees of the ERT algorithm are obtained, and the facial mean shape is obtained according to these regression trees.

11. The AU feature recognition method of claim 10, further comprising:

a judgment step;

judging whether probabilities of each AU feature in the prediction result are higher than the preset threshold value or not.

12. The AU feature recognition method of claim 11, wherein the judgment step further comprises:

a prompting step;

when it is judged that an AU feature of which the probability is higher than the preset threshold value exists in the prediction result, making such a prompt that the AU feature is recognized from the real-time facial image.

13. The AU feature recognition method of claim 9, wherein training steps for the AU classifier comprise:

a sample preparation step;

collecting a facial sample image, cropping an image region matched with each AU from the facial sample image as a positive sample image of the AU and preparing a negative sample image for each AU;

a local feature extraction step;

extracting local features of the positive sample image and negative sample image of each AU and generating corresponding feature vectors; and

a model training step;

performing learning training on a Support Vector Machine (SVM) by use of the local feature of the positive/negative sample image of each AU to obtain the corresponding AU classifier.

16. A computer-readable storage medium, in which an action unit (AU) feature recognition program is stored, the AU feature recognition program being executed by a processor to implement all steps of the AU feature recognition method of claim 9.

21. A computer-readable storage medium, in which an action unit (AU) feature recognition program is stored, the AU feature recognition program being executed by a processor to implement all steps of the AU feature recognition method of claim 10.

22. A computer-readable storage medium, in which an action unit (AU) feature recognition program is stored, the AU feature recognition program being executed by a processor to implement all steps of the AU feature recognition method of claim 11.

23. A computer-readable storage medium, in which an action unit (AU) feature recognition program is stored, the AU feature recognition program being executed by a processor to implement all steps of the AU feature recognition method of claim 12.

24. A computer-readable storage medium, in which an action unit (AU) feature recognition program is stored, the AU feature recognition program being executed by a processor to implement all steps of the AU feature recognition method of claim 13.

14-15. -15. (canceled)

17-20. -20. (canceled)