Predictive video analytics system and methods

• US 9,437,215 B2
• Filed: 01/15/2016
• Issued: 09/06/2016
• Est. Priority Date: 10/27/2014
• Status: Active Grant

• Current Assignee: Mattersight Corporation (Nice Ltd)
• Original Assignee: Mattersight Corporation (Nice Ltd)
• Inventors: Conway, Kelly, Danson, Christopher
• Primary Examiner(s): He, Jialong
• Application Number:
  US14/996,913
• Publication Number:
  US 20160133274A1
• Time in Patent Office: 235 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04M 2203/357 Autocues for dialog assistance
• CPC Class Codes: H04M 3/2281 Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls
• CPC Class Codes: H04M 3/51 Centralised call answering arrangements requiring operator intervention , e.g. call or contact centers for telemarketing

Abstract

The methods and systems described herein predict user behavior based on analysis of a user video communication. The methods include receiving a user video communication, extracting video facial analysis data from the video communication, extracting voice analysis data from the video communication, associating the video facial analysis data with the voice analysis data to determine an emotional state of a user, collecting biographical profile information specific to the user, applying a linguistic-based psychological behavioral model to the spoken words to determine personality type of the user, and inputting the collected biographical profile information, emotional state, and personality type into a predictive model to determine a likelihood of an outcome of the video communication.

Claims

1. A video analytics system adapted to predict user behavior based on analysis of a video communication, which comprises:

a node comprising a processor and a non-transitory computer readable medium operably coupled thereto, the non-transitory computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, where the plurality of instructions comprises;

instructions that, when executed, receive a video communication from a user, wherein the video communication comprises an audio component and a video component;

instructions that, when executed, analyze the video component to provide time-coded video behavioral data from the user;

instructions that, when executed, analyze the audio component to provide time-coded spoken words from the user;

instructions that, when executed, associate the time-coded spoken words with the video behavioral data to determine an emotional state of the user;

instructions that, when executed, collect biographical profile information specific to the user;

instructions that, when executed, determine a personality type of the user by applying a linguistic-based algorithm to the spoken words, searching a density of keywords in the spoken words, and comparing the keywords to a library separated by different personality types; and

instructions that, when executed, enter the collected biographical profile information, the emotional state, and the personality type into a predictive model, wherein the predictive model generates an indication of a likelihood of an outcome of the video communication.

2. The system of claim 1, wherein the outcome comprises one or more of whether a user will terminate his or her account, whether a user will purchase a product, whether a user is a fraudster, and whether a user will initiate additional subsequent interaction sessions regarding an issue.

3. The system of claim 1, wherein the video component includes a non-verbal, non-textual element comprising one or more of eye movement, facial expressions, gestures, activities, body postures, behaviors, attire, and actions.

4. The system of claim 1, wherein more than one user is speaking in the video communication.

5. The system of claim 4, wherein the audio and video components are analyzed simultaneously to determine which user is speaking.

6. The system of claim 1, which further comprises instructions, that when executed, identify attire of the user.

7. The system of claim 6, which further comprises instructions, that when executed, determine user value data based on estimated cost of the attire.

8. The system of claim 6, which further comprises instructions, that when executed, recommend items based on the attire of the user and display the items to an agent.

9. The system of claim 1, which further comprises instructions, that when executed, determine distress and engagement of the user.

10. The system of claim 9, wherein the distress and engagement are entered into the predictive model.

11. The system of claim 1, which further comprises instructions, that when executed, build the predictive model using previously-obtained emotional states and personality types of a plurality of different users.

12. The system of claim 11, wherein the emotional states are aggregated by personality types.

13. The system of claim 1, which further comprises instructions, that when executed:

provide the indication of the outcome to an agent;

determine, based at least in part on the indication of the likelihood of the outcome of the video communication, whether the agent should speak specific words, perform one or more specific actions, or both; and

provide the agent with the specific words, the one or more specific actions, or both.

14. The system of claim 1, which further comprises instructions, that when executed:

determine the likelihood of the outcome of the video communication should be increased or decreased;

select a second agent for transferring the video communication to, wherein the transferring of the video communication to the second agent is determined, using at least the predictive model, to increase or decrease the likelihood of the outcome occurring; and

transfer the video communication to the second agent.

15. A method to predict user behavior based on analysis of a video communication, which comprises:

receiving, by one or more processors, a user video communication;

extracting, by the one or more processors, video facial analysis data for the user from the video communication;

extracting, by the one or more processors, voice analysis data including spoken words from the user video communication;

associating, by the one or more processors, the video facial analysis data with the voice analysis data to determine an emotional state of the user;

collecting, by the one or more processors, biographical profile information specific to the user;

determining, by the one or more processors, a personality type of the user by applying a linguistic-based algorithm to the spoken words, searching a density of keywords in the spoken words, and comparing the keywords to a library separated by different personality types; and

entering, by the one or more processors, the collected biographical profile information, the emotional state, and the personality type into a predictive model, wherein the predictive model generates an indication of a likelihood of an outcome of the video communication.

16. The method of claim 15, wherein the outcome comprises one or more of whether a user will terminate his or her account, whether a user will pay an outstanding bill, whether a user is a fraudster, and whether a user will initiate additional subsequent interaction sessions regarding an issue.

17. The method of claim 15, which further comprises analyzing the user'"'"'s clothing and accessories.

18. The method of claim 17, which further comprises determining user value data based on the analysis of the user'"'"'s clothing and accessories.

19. The method of claim 15, which further comprises generating time-coded distress and engagement data for the user.

20. The method of claim 19, which further comprises aggregating the time-coded distress and engagement data with the video facial analysis data and the voice analysis data.

21. The method of claim 15, which further comprises displaying the outcome to an agent and providing the agent with specific words, specific actions, or both.

22. The method of claim 15, which further comprises automatically sending an email with a special offer to the user based on the outcome.

23. The method of claim 15, which further comprises generating an agent performance score or generating agent training materials based on the outcome.

24. A non-transitory machine-readable medium comprising instructions which, in response to a computer system, cause the computer system to perform a method which comprises:

receiving a user video communication;

separating an audio component from a video component of the video communication;

analyzing facial expressions of the user in the video component;

transcribing words spoken of the user in the audio component;

associating the facial expressions and spoken words to determine an emotional state of the user;

collecting biographical profile information specific to the user;

determining a personality type of the user by applying a linguistic-based algorithm to the spoken words, searching a density of keywords in the spoken words, and comparing the keywords to a library separated by different personality types; and

entering the collected biographical profile information, the emotional state, and the personality type into a predictive model, wherein the predictive model generates an indication of a likelihood of an outcome of the video communication.

25. The non-transitory machine-readable medium of claim 24, wherein the method further comprises aggregating the facial expressions and spoken words of a plurality of different users by personality type.

26. The non-transitory machine-readable medium of claim 25, wherein the method further comprises building a predictive model based on the aggregated facial expressions, spoken words, and personality type.

27. The non-transitory machine-readable medium of claim 24, wherein the method further comprises identifying the attire of the user.

28. The non-transitory machine-readable medium of claim 27, wherein the method further comprises aggregating the identified attire with the facial expressions and spoken words from a plurality of users.

29. The non-transitory machine-readable medium of claim 24, wherein the method further comprises determining distress and engagement data of the user.

30. The non-transitory machine-readable medium of claim 29, wherein the method further comprises aggregating the distress and engagement data with the facial expressions and spoken words.