Method and apparatus for recognizing facial expression

• US 10,387,716 B2
• Filed: 03/23/2017
• Issued: 08/20/2019
• Est. Priority Date: 10/21/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Youngsung, Yoo, Byungin, Kwak, Youngjun, Choi, Chang Kyu
• Primary Examiner(s): Hon, Ming Y
• Application Number:
  US15/467,396
• Publication Number:
  US 20180114057A1
• Time in Patent Office: 880 Days
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression

Abstract

A method and apparatus for recognizing a facial expression is provided. The method of recognizing the facial expression includes extracting a feature from an input image, extracting a feature from a reference image, and recognizing a user facial expression based on a difference between the feature of the input image and the feature of the reference image.

Claims

1. A processor-implemented method of recognizing a facial expression of a user, the method comprising:

extracting a first feature from an input image, the first feature corresponding to a first expression represented in the input image;

generating a reference image based on the input image;

obtaining a second feature from the reference image, the second feature corresponding to a second expression represented in the reference image;

determining a difference between the first expression and the second expression based on the first feature and the second feature; and

recognizing the facial expression based on the determined difference between the first expression and the second expression.

2. The method of claim 1, wherein the first feature of the input image and the second feature of the reference image are respectively extracted and obtained by extractors that are trained in advance.

3. The method of claim 2, wherein the extractors are trained in advance to increase the difference between the first feature of the input image and the second feature of the reference image within a threshold.

4. The method of claim 2, wherein the extractors share a parameter.

5. The method of claim 2, wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the first feature of the input image, and to decrease a difference between a second image converted based on the second feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

6. The method of claim 5, wherein the extractors are trained in advance to increase a difference between the first image and the second image within a threshold.

7. The method of claim 1, wherein the obtaining of the second feature from the reference image comprises obtaining the second feature from a first reference image and a third feature from a second reference image, and the performing of the recognition of the user facial expression comprises recognizing the user facial expression based on a difference between the first feature of the input image and the second feature, and a difference between the first feature of the input image and the third feature.

8. The method of claim 1, wherein the difference comprises an element-wise difference between a feature vector, corresponding to the first feature, extracted from the input image and a feature vector, corresponding to the second feature, extracted from the reference image.

9. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, causes the processor to perform the method of claim 1.

36. The method of claim 1, wherein the reference image is generated based on the input image using a trained neural network based statistical model.

10. A processor-implemented method of recognizing a facial expression of a user, the method comprising:

extracting a feature from an input image;

encoding the input image using an encoder that is trained in advance; and

generating a reference image by decoding the encoded input image using a decoder that is trained in advance;

extracting a feature from the reference image; and

recognizing the facial expression based on a difference between the feature of the input image and the feature of the reference image.

11. The method of claim 10, wherein the encoder and the decoder are trained in advance to decrease a difference between the reference image and a target image corresponding to a facial expression to be represented through the reference image.

12. The method of claim 11, wherein the encoder and the decoder are trained in advance to increase a difference between the input image and the reference image within a threshold.

13. A facial expression recognizing apparatus comprising:

a processor configured

to extract from a input image a first feature that corresponds to a first expression represented in the input image,

to generate a reference image based on the input image;

to obtain from a reference image a second feature that corresponds to a second expression represented in the reference image,

to determine a difference between the first expression and the second expression based on the first feature and the second feature, and

to recognize a facial expression of a user based on the determined difference between the first expression and the second expression.

16. The apparatus of claim 13, wherein the reference image is generated based on the input image using a trained neural network based statistical model.

17. The apparatus of claim 13 wherein the feature of the input image and the feature of the reference image are extracted by extractors that are trained in advance.

18. The apparatus of claim 17, wherein the extractors are trained in advance to increase a difference between the feature of the input image and the feature of the reference image within a threshold.

19. The apparatus of claim 17, wherein the extractors share a parameter.

20. The apparatus of claim 17, wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the feature of the input image, and to decrease a difference between a second image converted based on the feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

21. The apparatus of claim 20, wherein the first image and the second image are converted by converters that are trained in advance, and the converters are trained in advance to increase a difference between the first image and the second image within a threshold.

22. The apparatus of claim 13, further comprising a memory configured to store instructions, which when executed by the processor configure the processor to perform the extraction of the first feature, the obtaining of the second features, the determination of the difference, and the recognition of the facial expression of the user.

23. The apparatus of claim 13,

wherein the reference image comprises a first reference image and a second reference image, and

wherein the processor is configured to obtain the second feature from the first reference image and a third feature from the second reference image, and to recognize the user facial expression based on a difference between the first feature of the input image and the second feature and a difference between the first feature of the input image and the third feature.

14. A processor-implemented method of recognizing a facial expression of a user, the method comprising:

extracting a first feature from an input image, the first feature corresponding to a first expression represented in the input image;

obtaining a second feature from a reference image, the second feature corresponding to a second expression represented in the reference image;

determining a difference between the first expression and the second expression based on the first feature and the second feature; and

recognizing the facial expression based on the determined difference between the first expression and the second expression,

wherein the first feature of the input image and the second feature of the reference image are respectively extracted and obtained by extractors that are trained in advance, and

wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the first feature of the input image, and to decrease a difference between a second image converted based on the second feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

15. The method of claim 14, wherein the extractors are trained in advance to increase a difference between the first image and the second image within a threshold.

24. A facial expression recognizing apparatus comprising:

a processor configured to extract a feature from an input image, encode the input image using an encoder that is trained in advance, generate a reference image by decoding the encoded input image using a decoder that is trained in advance, extract a feature from the reference image, and recognize a facial expression of a user based on a difference between the feature of the input image and the feature of the reference image.

25. The apparatus of claim 24, wherein the encoder and the decoder are trained in advance to decrease a difference between the reference image and a target image corresponding to a facial expression to be represented through the reference image.

26. The apparatus of claim 25, wherein the encoder and the decoder are trained in advance to decrease a difference between the input image and the reference image within a threshold.

27. The apparatus of claim 24, wherein the feature of the input image and the feature of the reference image are extracted by extractors that are trained in advance.

28. The apparatus of claim 27, wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the feature of the input image, and to decrease a difference between a second image converted based on the feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

29. The apparatus of claim 28, wherein the first image and the second image are converted by converters that are trained in advance, and the converters are trained in advance to increase a difference between the first image and the second image within a threshold.

30. The apparatus of claim 27, wherein the extractors are trained in advance to increase a difference between the feature of the input image and the feature of the reference image within a threshold.

31. The apparatus of claim 27, wherein the extractors share a parameter.

32. The apparatus of claim 24, wherein the reference image comprises a first reference image and a second reference image, and the processor is configured to perform the extraction of the feature from the reference image by extraction of a first feature from the first reference image and a second feature from the second reference image, and to perform the recognition of the facial expression based on a difference between the feature of the input image and the first feature and a difference between the feature of the input image and the second feature.

33. The apparatus of claim 24, further comprising a memory configured to store instructions, that when executed by the processor configure the processor to perform the extraction of the feature from the input image, encode the input image using the encoder, generate the reference image, extract the feature from the reference image, and recognize the facial expression of the user.

34. A facial expression recognizing apparatus comprising:

a processor configured to extract a feature from an input image, to obtain a feature from a reference image, and to recognize a facial expression of a user based on a difference between the feature of the input image and the feature of the reference image,

wherein the feature of the input image and the feature of the reference image are respectively extracted and obtained by extractors that are trained in advance, and

wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the feature of the input image, and to decrease a difference between a second image converted based on the feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

35. The apparatus of claim 34, wherein the first image and the second image are converted by converters that are trained in advance, and the converters are trained in advance to increase a difference between the first image and the second image within a threshold.

37. A facial expression recognizing apparatus comprising:

a processor configured to extract a feature from an input image, to obtain a feature from a reference image, and to recognize a facial expression of a user based on a difference between the feature of the input image and the feature of the reference image,

wherein the reference image comprises a first reference image and a second reference image respectively generated based on the input image, and the processor is further configured to obtain a first feature from the first reference image and a second feature from the second reference image, and to recognize the user facial expression based on a difference between the feature of the input image and the first feature and a difference between the feature of the input image and the second feature.

38. A digital device comprising:

a sensor configured to capture an input image;

a memory configured to store instructions; and

a processor configured to execute the instructions to extract from the input image a first feature that corresponds to a first expression represented in the input image, generate a reference image based on the input image, to obtain from the reference image a second feature that corresponds to a second expression represented in the reference image, to determine a difference between the first expression and the second expression based on the first feature and the second feature, and to recognize a facial expression of a user based on the determined difference between the first expression and the second expression.

39. The digital device of claim 38, wherein the processor comprises:

one or more extractors configured to perform the extraction of the first feature from the input image, and perform the obtaining of the second feature to obtain the second feature from the reference image;

a comparer configured to perform the determination of the difference by comparing the first feature of the input image with the second feature of the reference image; and

a recognizer configured to perform the recognition of the facial expression of the input image based on a result of the comparison.

40. The digital device of claim 38, further comprising a display, and wherein the processor is further configured to display information of the recognized facial expression of the user on the display.

41. The digital device of claim 38, further comprising:

an antenna; and

a cellular radio configured to transmit and receive data via the antenna according to a cellular communications standard,

wherein the processor is further configured to transmit information of the recognized facial expression of the user using the cellular radio.

42. The digital device of claim 38, wherein the reference image is generated based on the input image using a trained neural network based statistical model.

43. A facial expression recognizing apparatus comprising:

a processor configured to extract from a input image a first feature that corresponds to a first expression represented in the input image, to obtain from a reference image a second feature that corresponds to a second expression represented in the reference image, to determine a difference between the first expression and the second expression based on the first feature and the second feature, and to recognize a facial expression of a user based on the determined difference between the first expression and the second expression,

wherein the feature of the input image and the feature of the reference image are extracted by extractors that are trained in advance, and

wherein the extractors are trained in advance to decrease a difference between the input image and a first image converted based on the feature of the input image, and to decrease a difference between a second image converted based on the feature of the reference image and a target image corresponding to a facial expression to be represented through the reference image.

44. The apparatus of claim 43, wherein the first image and the second image are converted by converters that are trained in advance, and the converters are trained in advance to increase a difference between the first image and the second image within a threshold.