Emotion recognition for workforce analytics

• US 9,747,573 B2
• Filed: 03/23/2015
• Issued: 08/29/2017
• Est. Priority Date: 03/23/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Avatar Merger Sub II, LLC
• Inventors: Shaburov, Victor, Monastyrshin, Yurii
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US14/665,686
• Publication Number:
  US 20150193718A1
• Time in Patent Office: 890 Days
• Field of Search: 382103, 382118, 382168, 382173, 382181, 382190, 382195, 382209, 382224, 382254, 382305, 706 47, 706 55, 706 59, 705 739
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/06395 Quality analysis or management
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04N 21/440218 by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4 conversion of standards in analogue television systems H04N7/01
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Methods and systems for videoconferencing include generating work quality metrics based on emotion recognition of an individual such as a call center agent. The work quality metrics allow for workforce optimization. One example method includes the steps of receiving a video including a sequence of images, detecting an individual in one or more of the images, locating feature reference points of the individual, aligning a virtual face mesh to the individual in one or more of the images based at least in part on the feature reference points, dynamically determining over the sequence of images at least one deformation of the virtual face mesh, determining that the at least one deformation refers to at least one facial emotion selected from a plurality of reference facial emotions, and generating quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

Claims

1. A computer-implemented method for workforce analytics, the method comprising:

receiving a set of videos including a sequence of images, each video of the set of videos including at least one individual;

for each video of the set of videos, detecting a respective individual in one or more of the images;

locating, within each video of the set of videos, feature reference points of the respective individual;

aligning, within each video of the set of videos, a virtual face mesh to the respective individual based at least in part on the feature reference points;

dynamically determining, for each video of the set of videos, over the sequence of images at least one deformation of the virtual face mesh;

determining, for each video of the set of videos, that the at least one deformation refers to at least one facial emotion of the respective individual within the video, the at least one facial emotion identified as matching a facial emotion from a plurality of reference facial emotions;

identifying, for a predetermined period of time commonly represented within the set of videos, a set of facial emotions for the respective individuals within the set of videos;

identifying an emotional status for a target user for the predetermined period of time, the target user interacting with the individuals included in the set of videos, and the emotional status determined based on the set of facial emotions of the individuals within the set of videos; and

generating quality metrics for the target user, the quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

2. The method of claim 1, further comprising recording the quality metrics of the individual in an employee record, wherein each of the quality metrics is time-stamped.

3. The method of claim 1, further comprising aggregating the quality metrics associated with the individual over a predetermined period to produce a work performance characteristic of the individual.

4. The method of claim 1, wherein the east one work quality parameter includes a tiredness characteristic of the individual.

5. A system, comprising:

a computing device including at least one processor and a memory storing processor-executable codes, which, when implemented by the at least one processor, cause to perform the steps of;

receiving a set of videos including a sequence of images, each video of the set of videos including at least one individual;

for each video of the set of videos, detecting a respective individual in one or more of the images;

locating, within each video of the set of videos, feature reference points of the respective individual;

aligning, within each video of the set of videos, a virtual face mesh to the respective individual based at least in part on the feature reference points;

dynamically determining, for each video of the set of videos, over the sequence of images at least one deformation of the virtual face mesh;

determining, for each video of the set of videos, that the at least one deformation refers to at least one facial emotion of the respective individual within the video, the at least one facial emotion identified as matching a facial emotion from a plurality of reference facial emotions;

identifying, for a predetermined period of time commonly represented within the set of videos, a set of facial emotions for the respective individuals within the set of videos;

identifying an emotional status for a target user for the predetermined period of time, the target user interacting with the individuals included in the set of videos, and the emotional status determined based on the set of facial emotions of the individuals within the set of videos; and

generating quality metrics for the target user, the quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.

6. A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method, comprising:

receiving a set of videos including a sequence of images, each video of the set of videos including at least one individual;

for each video of the set of videos, detecting a respective individual one or more of the images;

locating, within each video of the set of videos, feature reference points of the respective individual;

aligning, within each video of the set of videos, a virtual face mesh to the respective individual based at least in part on the feature reference points;

dynamically determining, for each video of the set of videos, over the sequence of images at least one deformation of the virtual face mesh;

determining, for each video of the set of videos, that the at least one deformation refers to at least one facial emotion of the respective individual within the video, the at least one facial emotion identified as matching a facial emotion from a plurality of reference facial emotions;

identifying, for a predetermined period of time commonly represented within the set of videos, a set of facial emotions for the respective individuals within the set of videos;

identifying an emotional status for a target user for the predetermined period of time, the target user interacting with the individuals included in the set of videos, and the emotional status determined based on the set of facial emotions of the individuals within the set of videos; and

generating quality metrics for the target user, the quality metrics including at least one work quality parameter associated with the individual based on the at least one facial emotion.