METHOD AND APPARATUS FOR RECOGNIZING FACIAL EXPRESSION

• US 20180114057A1
• Filed: 03/23/2017
• Published: 04/26/2018
• Est. Priority Date: 10/21/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: KIM, Youngsung, YOO, Byungin, KWAK, Youngjun, CHOI, Chang Kyu
• Granted Patent: US 10,387,716 B2
• Time in Patent Office: Days
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A method and apparatus for recognizing a facial expression is provided. The method of recognizing the facial expression includes extracting a feature from an input image, extracting a feature from a reference image, and recognizing a user facial expression based on a difference between the feature of the input image and the feature of the reference image.

Claims

1. A method of recognizing a facial expression of a user, the method comprising:

extracting a feature from an input image;

extracting a feature from a reference image; and

recognizing the facial expression based on a difference between the feature of the input image and the feature of the reference image.

2. The method of claim 1, further comprising:

generating the reference image based on the input image.

3. The method of claim 1, further comprising:

encoding the input image using an encoder that is trained in advance; and

generating the reference image by decoding the encoded input image using a decoder that is trained in advance.

4. The method of claim 3, wherein the encoder and the decoder are trained in advance to decrease a difference between the reference image and a target image corresponding to a facial expression to be represented through the reference image.

5. The method of claim 4, wherein the encoder and the decoder are trained in advance to increase a difference between the input image and the reference image within a threshold.

6. The method of claim 1, wherein the feature of the input image and the feature of the reference image are extracted by extractors that are trained in advance.

7. The method of claim 6, wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the feature of the input image, and to decrease a difference between a second image converted based on the feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

8. The method of claim 7, wherein the extractors are trained in advance to increase a difference between the first image and the second image within a threshold.

9. The method of claim 6, wherein the extractors are trained in advance to increase the difference between the feature of the input image and the feature of the reference image within a threshold.

10. The method of claim 6, wherein the extractors share a parameter.

11. The method of claim 1, wherein the extracting of the feature from the reference image comprises extracting a first feature from a first reference image and a second feature from a second reference image, and the recognizing of the user facial expression comprises recognizing the user facial expression based on a difference between the feature of the input image and the first feature, and a difference between the feature of the input image and the second feature.

12. The method of claim 1, wherein the difference comprises an element-wise difference between a feature vector extracted from the input image and a feature vector extracted from the reference image.

13. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, causes the processor to perform the method of claim 1.

14. A facial expression recognizing apparatus comprising:

a processor configured to extract a feature from an input image, to extract a feature from a reference image, and to recognize a facial expression of a user based on a difference between the feature of the input image and the feature of the reference image.

15. The apparatus of claim 14, wherein the processor is further configured to generate the reference image based on the input image.

16. The apparatus of claim 14, wherein the processor is further configured to encode the input image using an encoder that is trained in advance and to generate the reference image by decoding the encoded input image using a decoder that is trained in advance.

17. The apparatus of claim 16, wherein the encoder and the decoder are trained in advance to decrease a difference between the reference image and a target image corresponding to a facial expression to be represented through the reference image.

18. The apparatus of claim 17, wherein the encoder and the decoder are trained in advance to decrease a difference between the input image and the reference image within a threshold.

19. The apparatus of claim 14, wherein the feature of the input image and the feature of the reference image are extracted by extractors that are trained in advance.

20. The apparatus of claim 19, wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the feature of the input image, and to decrease a difference between a second image converted based on the feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

21. The apparatus of claim 20, wherein the first image and the second image are converted by converters that are trained in advance, and the converters are trained in advance to increase a difference between the first image and the second image within a threshold.

22. The apparatus of claim 19, wherein the extractors are trained in advance to increase a difference between the feature of the input image and the feature of the reference image within a threshold.

23. The apparatus of claim 19, wherein the extractors share a parameter.

24. The apparatus of claim 14, wherein the reference image comprises a first reference image and a second reference image, and the processor is further configured to extract a first feature from the first reference image and a second feature from the second reference image, and to recognize the user facial expression based on a difference between the feature of the input image and the first feature and a difference between the feature of the input image and the second feature.

25. The sentence generating apparatus of claim 14, further comprising a memory configured to store instructions to be executed by the processor.

26. A digital device comprising:

a sensor configured to capture an input image;

a memory configured to store instructions; and

a processor configured to execute the instructions to extract a feature from the input image, to extract a feature from a reference image, and to recognize a facial expression of a user based on a difference between the feature of the input image and the feature of the reference image.

27. The digital device of claim 26, wherein the processor comprises:

an extractor configured to extract the feature from the input image, and to extract the feature from the reference image;

a comparer configured to compare the feature of the input image with the feature of the reference image; and

a recognizer configured to recognize the facial expression of the input image based on the comparison.

28. The digital device of claim 26, further comprising a display, and wherein the processor is further configured to display the facial expression of the user on the display.

29. The digital device of claim 26, further comprising:

an antenna; and

a cellular radio configured to transmit and receive data via the antenna according to a cellular communications standard,

wherein the processor is further configured to transmit the facial expression of the user using the cellular radio.