EMOTION RECOGNITION FOR WORKFORCE ANALYTICS

• US 20150193718A1
• Filed: 03/23/2015
• Published: 07/09/2015
• Est. Priority Date: 03/23/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Looksery, Inc. (Snap, Inc.)
• Inventors: Shaburov, Victor, Monastyrshin, Yurii
• Granted Patent: US 9,747,573 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04N 21/440218 by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 conversion of standards in analogue television systems H04N7/01
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Methods and systems for videoconferencing include generating work quality metrics based on emotion recognition of an individual such as a call center agent. The work quality metrics allow for workforce optimization. One example method includes the steps of receiving a video including a sequence of images, detecting an individual in one or more of the images, locating feature reference points of the individual, aligning a virtual face mesh to the individual in one or more of the images based at least in part on the feature reference points, dynamically determining over the sequence of images at least one deformation of the virtual face mesh, determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions, and generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

Claims

1. A computer-implemented method for workforce analytics, the method comprising:

receiving a video including a sequence of images;

detecting an individual in one or more of the images;

locating feature reference points of the individual;

aligning a virtual face mesh to the individual in one or more of the images based at least in part on the feature reference points;

dynamically determining over the sequence of images at least one deformation of the virtual face mesh;

determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions; and

generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

2. The method of claim 1, further comprising recording the quality metrics of the individual in an employee record, wherein each of the quality metrics is time-stamped.

3. The method of claim 1, further comprising establishing a video conference between the individual and a customer.

4. The method of claim 1, further comprising aggregating the quality metrics associated with the individual over a predetermined period to produce a work performance characteristic of the individual.

5. The method of claim 1, wherein the at least one work quality parameter includes a tiredness characteristic of the individual.

6. The method of claim 1, wherein the at least one work quality parameter includes a negative emotion characteristic of the individual.

7. The method of claim 1, wherein the at least one work quality parameter includes a positive emotion characteristic of the individual.

8. The method of claim 1, wherein the at least one work quality parameter includes a smile characteristic of the individual.

9. The method of claim 1, wherein the determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions includes:

comparing the at least one deformation of the virtual face mesh to reference facial parameters of the plurality of reference facial emotions; and

selecting the facial emotion based on the comparison of the at least one deformation of the virtual face mesh to the reference facial parameters of the plurality of reference facial emotions.

10. The method of claim 9, wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a convolution neural network.

11. The method of claim 9, wherein the comparing of the at least one deformation of the virtual face mesh to reference facial parameters comprises applying a state vector machine.

12. The method of claim 1, wherein the feature reference points include facial landmarks.

13. The method of claim 1, wherein the detecting of the individual includes applying a Viola-Jones algorithm to the images.

14. The method of claim 1, wherein the locating of the feature reference points includes applying an Active Shape Model algorithm areas of the images associated with the individual.

15. The method of claim 1, wherein the aligning of the virtual face mesh is based on shape units associated with a face shape of the individual.

16. The method of claim 15, further comprising:

estimating intensities of the shape units associated with the face shape;

estimating intensities of action units associated with face mimics; and

estimating rotations of the virtual face mesh around three orthogonal axes and its translations along the axes.

17. The method of claim 1, wherein the plurality of facial emotions include at least one of the following:

a neutral facial emotion, a positive facial emotion, and a negative facial emotion;

wherein the positive facial emotion includes at least one of happiness, gratitude, kindness, and enthusiasm; and

wherein the negative facial emotion includes at least one of anger, stress, depression, frustration, embarrassment, irritation, sadness, indifference, confusion, and annoyance.

18. The method of claim 1, further comprising:

receiving an audio stream associated with the video; and

recognizing a speech emotion of the individual in the audio stream.

19. The method of claim 18, wherein the recognizing of the speech emotion comprising:

extracting at least one voice feature from the audio stream;

comparing the extracted at least one voice feature to a plurality of reference voice features; and

selecting the speech emotion based on the comparison of the extracted at least one voice feature to the plurality of reference voice features.

20. The method of claim 18, wherein the recognizing of the speech emotion comprises recognizing a speech in the audio stream.

21. The method of claim 18, further comprising combining the facial emotion and the speech emotion to generate the at least one work quality parameter of the individual.

22. A system, comprising:

a computing device including at least one processor and a memory storing processor-executable codes, which, when implemented by the at least one processor, cause to perform the steps of;

receiving a video including a sequence of images;

detecting an individual in one or more of the images;

locating feature reference points of the individual;

aligning a virtual face mesh to the individual in one or more of the images based at least in part on the feature reference points;

dynamically determining over the sequence of images at least one deformation of the virtual face mesh;

determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions; and

generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

23. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method, comprising:

receiving a video including a sequence of images;

detecting an individual in one or more of the images;

locating feature reference points of the individual;

aligning a virtual face mesh to the individual in one or more of the images based at least in part on the feature reference points;

dynamically determining at least one deformation of the virtual face mesh;

determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions; and

generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.