Emotion recognition in video conferencing

• US 9,852,328 B2
• Filed: 02/10/2017
• Issued: 12/26/2017
• Est. Priority Date: 03/18/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Shaburov, Victor, Monastyrshyn, Yurii
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US15/430,133
• Publication Number:
  US 20170154211A1
• Time in Patent Office: 319 Days
• Field of Search: None
• CPC Class Codes: G06Q 30/0281 Customer communication at a business location, e.g. providing product or service information, consulting
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Methods and systems for videoconferencing include recognition of emotions related to one videoconference participant such as a customer. This ultimately enables another videoconference participant, such as a service provider or supervisor, to handle angry, annoyed, or distressed customers. One example method includes the steps of receiving a video that includes a sequence of images, detecting at least one object of interest (e.g., a face), locating feature reference points of the at least one object of interest, aligning a virtual face mesh to the at least one object of interest based on the feature reference points, finding over the sequence of images at least one deformation of the virtual face mesh that reflect face mimics, determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions, and generating a communication bearing data associated with the facial emotion.

Claims

1. A computer-implemented method for video conferencing, the method comprising:

receiving a video including a sequence of images and an audio stream;

detecting at least one object of interest in one or more of the images;

locating feature reference points of the at least one object of interest;

aligning a virtual face mesh to the at least one object of interest in one or more of the images based at least in part on the feature reference points;

finding over the sequence of images at least one deformation of the virtual face mesh, wherein the at least one deformation is associated with at least one face mimic;

determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions;

recognizing a speech emotion in the audio stream of the at least one object of interest; and

generating a communication bearing data associated with one or more of the facial emotion and the speech emotion.

2. The computer-implemented method of claim 1, wherein recognizing the speech emotion comprises:

extracting at least one voice feature from the audio stream;

comparing the extracted at least one voice feature to a plurality of reference voice features; and

selecting the speech emotion based on the comparison of the extracted at least one voice feature to the plurality of reference voice features.

3. The computer-implemented method of claim 1, wherein the object of interest is a first user and the video stream comprises speech from the first user and a second user, and the method further comprises:

recognizing a speech emotion of the second user; and

recognizing the speech emotion of the first user based on the speech of the first user and the speech emotion of the second user.

4. The computer-implemented method of claim 1, wherein recognizing the speech emotion comprises recognizing a speech in the audio stream.

5. The computer-implemented method of claim 1, wherein the communication bearing data associated with the facial emotion further includes data associated with the speech emotion.

6. The computer-implemented method of claim 1 further comprising:

combining the facial emotion and the speech emotion to generate an emotional status of an individual associated with the at least one object of interest.

7. The computer-implemented method of claim 1, wherein recognizing the speech emotion further comprises:

identifying one or more keywords within the audio stream of the at least one object of interest;

determining at least one keyword of the one or more keywords is associated with a negative emotion; and

selecting a negative emotion as the speech emotion of the at least one object of interest.

8. A system, comprising:

one or more processors; and

a non-transitory processor-readable medium coupled to the one or more processors, the non-transitory processor-readable medium comprising processor-executable instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising;

receiving a video including a sequence of images and an audio stream;

detecting at least one object of interest in one or more of the images;

locating feature reference points of the at least one object of interest;

aligning a virtual face mesh to the at least one object of interest in one or more of the images based at least in part on the feature reference points;

finding over the sequence of images at least one deformation of the virtual face mesh, wherein the at least one deformation is associated with at least one face mimic;

determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions;

recognizing a speech emotion in the audio stream of the at least one object of interest; and

generating a communication bearing data associated with one or more of the facial emotion and the speech emotion.

9. The system of claim 8, wherein recognizing the speech emotion comprises:

extracting at least one voice feature from the audio stream;

comparing the extracted at least one voice feature to a plurality of reference voice features; and

selecting the speech emotion based on the comparison of the extracted at least one voice feature to the plurality of reference voice features.

10. The system of claim 8, wherein the object of interest is a first user and the video stream comprises speech from the first user and a second user, and the operations further comprise:

recognizing a speech emotion of the second user; and

recognizing the speech emotion of the first user based on the speech of the first user and the speech emotion of the second user.

11. The system of claim 8, wherein recognizing the speech emotion comprises recognizing a speech in the audio stream.

12. The system of claim 8, wherein the communication bearing data associated with the facial emotion further includes data associated with the speech emotion.

13. The system of claim 8, wherein the operations further comprise:

combining the facial emotion and the speech emotion to generate an emotional status of an individual associated with the at least one object of interest.

14. The system of claim 8, wherein recognizing the speech emotion further comprises:

identifying one or more keywords within the audio stream of the at least one object of interest;

determining at least one keyword of the one or more keywords is associated with a negative emotion; and

selecting a negative emotion as the speech emotion of the at least one object of interest.

15. A non-transitory processor-readable medium comprising processor-executable instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving a video including a sequence of images and an audio stream;

detecting at least one object of interest in one or more of the images;

locating feature reference points of the at least one object of interest;

aligning a virtual face mesh to the at least one object of interest in one or more of the images based at least in part on the feature reference points;

finding over the sequence of images at least one deformation of the virtual face mesh, wherein the at least one deformation is associated with at least one face mimic;

determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions;

recognizing a speech emotion in the audio stream of the at least one object of interest; and

generating a communication bearing data associated with one or more of the facial emotion and the speech emotion.

16. The non-transitory processor-readable medium of claim 15, wherein recognizing the speech emotion comprises:

extracting at least one voice feature from the audio stream;

comparing the extracted at least one voice feature to a plurality of reference voice features; and

selecting the speech emotion based on the comparison of the extracted at least one voice feature to the plurality of reference voice features.

17. The non-transitory processor-readable medium of claim 15, wherein the object of interest is a first user and the video stream comprises speech from the first user and a second user, and the operations further comprise:

recognizing a speech emotion of the second user; and

recognizing the speech emotion of the first user based on the speech of the first user and the speech emotion of the second user.

18. The non-transitory processor-readable medium of claim 15, wherein recognizing the speech emotion comprises recognizing a speech in the audio stream.

19. The non-transitory processor-readable medium of claim 15, wherein the communication bearing data associated with the facial emotion further includes data associated with the speech emotion.

20. The non-transitory processor-readable medium of claim 15, wherein the operations further comprise:

combining the facial emotion and the speech emotion to generate an emotional status of an individual associated with the at least one object of interest.