Emotion recognition for workforce analytics

• US 10,496,947 B1
• Filed: 08/28/2017
• Issued: 12/03/2019
• Est. Priority Date: 03/23/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Shaburov, Victor, Monastyrshyn, Yurii
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US15/688,480
• Time in Patent Office: 827 Days
• Field of Search: 382103, 382118, 382168, 382173, 382181, 382190, 382195, 382209, 382224, 382254, 382305, 706 47, 706 55, 706 59, 705 739
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04N 21/440218 by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 conversion of standards in analogue television systems H04N7/01
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Methods and systems for videoconferencing include generating work quality metrics based on emotion recognition of an individual such as a call center agent. The work quality metrics allow for workforce optimization. One example method includes the steps of receiving a video including a sequence of images, detecting an individual in one or more of the images, locating feature reference points of the individual, aligning a virtual face mesh to the individual in one or more of the images based at least in part on the feature reference points, dynamically determining over the sequence of images at least one deformation of the virtual face mesh, determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions, and generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

Claims

1. A computer-implemented method for workforce analytics, the method comprising:

receiving a video stream including a sequence of frames;

detecting an individual in one or more of the frames;

locating feature reference points of the individual;

aligning a virtual face mesh to the individual in one or more of the frames based at least in part on the feature reference points;

dynamically determining over the sequence of frames at least one deformation of the virtual face mesh;

determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions;

identifying an emotional status for the individual, the emotional status determined based on the at least one facial emotion of the individual within the video stream; and

generating at least one work quality parameter associated with the individual based on the at least one facial emotion and the emotional status.

2. The method of claim 1, further comprising establishing a video conference between the individual and a customer.

3. The method of claim 1, wherein the at least one work quality parameter includes a negative emotion characteristic of the individual.

4. The method of claim 1, wherein the at least one work quality parameter includes a positive emotion characteristic of the individual.

5. The method of claim 1, wherein the at least one work quality parameter includes a smile characteristic of the individual.

6. The method of claim 1, wherein the determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions includes:

comparing the at least one deformation of the virtual face mesh to reference facial parameters of the plurality of reference facial emotions; and

selecting the facial emotion based on the comparison of the at least one deformation of the virtual face mesh to the reference facial parameters of the plurality of reference facial emotions.

7. The method of claim 6, wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a convolution neural network.

8. The method of claim 6, wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a state vector machine.

9. The method of claim 1, wherein the feature reference points include facial landmarks.

10. The method of claim 1, wherein the detecting of the individual includes applying a Viola-Jones algorithm to the frames.

11. The method of claim 1, wherein the locating of the feature reference points includes applying an Active Shape Model algorithm areas of the frames associated with the individual.

12. The method of claim 1, wherein the aligning of the virtual face mesh is based on shape units associated with a face shape of the individual.

13. The method of claim 12, further comprising:

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics; and

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes.

14. The method of claim 1, wherein the plurality of facial emotions include at least one of the following:

a neutral facial emotion, a positive facial emotion, and a negative facial emotion;

wherein the positive facial emotion includes at least one of happiness, gratitude, kindness, and enthusiasm; and

wherein the negative facial emotion includes at least one of anger, stress, depression, frustration, embarrassment, irritation, sadness, indifference, confusion, and annoyance.

15. The method of claim 1, further comprising:

receiving an audio stream associated with the video stream; and

recognizing a speech emotion of the individual in the audio stream.

16. The method of claim 15, wherein the recognizing of the speech emotion comprising:

extracting at least one voice feature from the audio stream;

comparing the extracted at least one voice feature to a plurality of reference voice features; and

selecting the speech emotion based on the comparison of the extracted at least one voice feature to the plurality of reference voice features.

17. The method of claim 15, wherein the recognizing of the speech emotion comprises recognizing a speech in the audio stream.

18. The method of claim 15, further comprising combining the facial emotion and the speech emotion to generate the at least one work quality parameter of the individual.

19. A system, comprising:

a computing device including at least one processor and a memory storing processor-executable codes, which, when implemented by the at least one processor, cause to perform operations comprising;

receiving a video stream including a sequence of frames;

detecting an individual in one or more of the frames;

locating feature reference points of the individual;

aligning a virtual face mesh to the individual in one or more of the frames based at least in part on the feature reference points;

dynamically determining over the sequence of frames at least one deformation of the virtual face mesh;

determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions;

identifying an emotional status for the individual, the emotional status determined based on the at least one facial emotion of the individual within the video stream; and

generating at least one work quality parameter associated with the individual based on the at least one facial emotion and the emotional status.

20. The system of claim 19, wherein determining that the at least one deformation refers to at least one facial emotion further comprises:

comparing the at least one deformation of the virtual face mesh to reference facial parameters of the plurality of reference facial emotions; and

selecting the facial emotion based on the comparison of the at least one deformation of the virtual face mesh to the reference facial parameters of the plurality of reference facial emotions.

21. The system of claim 19, wherein the operations further comprise:

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics; and

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes.

22. The system of claim 19, wherein the operations further comprise:

receiving an audio stream associated with the video; and

recognizing a speech emotion of the individual in the audio stream.

23. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to perform operations, comprising:

receiving a video stream including a sequence of frames;

detecting an individual in one or more of the frames;

locating feature reference points of the individual;

aligning a virtual face mesh to the individual in one or more of the frames based at least in part on the feature reference points;

dynamically determining at least one deformation of the virtual face mesh;

determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions;

identifying an emotional status for the individual, the emotional status determined based on the at least one facial emotion of the individual within the video stream; and

generating at least one work quality parameter associated with the individual based on the at least one facial emotion and the emotional status.

24. The non-transitory processor-readable medium of claim 23, wherein determining that the at least one deformation refers to at least one facial emotion further comprises:

comparing the at least one deformation of the virtual face mesh to reference facial parameters of the plurality of reference facial emotions; and

selecting the facial emotion based on the comparison of the at least one deformation of the virtual face mesh to the reference facial parameters of the plurality of reference facial emotions.

25. The non-transitory processor-readable medium of claim 23, wherein the operations further comprise:

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics; and

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes.

26. The non-transitory processor-readable medium of claim 23, wherein the operations further comprise:

receiving an audio stream associated with the video; and

recognizing a speech emotion of the individual in the audio stream.