Predictive and responsive video analytics system and methods

• US 10,262,195 B2
• Filed: 08/23/2016
• Issued: 04/16/2019
• Est. Priority Date: 10/27/2014
• Status: Active Grant

• Current Assignee: Mattersight Corporation (Nice Ltd)
• Original Assignee: Mattersight Corporation (Nice Ltd)
• Inventors: Conway, Kelly, Danson, Christopher
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/244,794
• Publication Number:
  US 20160364606A1
• Time in Patent Office: 966 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/57 for processing of video signals
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04M 2203/357 Autocues for dialog assistance
• CPC Class Codes: H04M 3/2281 Call monitoring, e.g. for law enforcement purposes; Call tracing; Detection or prevention of malicious calls
• CPC Class Codes: H04M 3/51 Centralised call answering arrangements requiring operator intervention , e.g. call or contact centers for telemarketing

Abstract

Methods and systems to predict user behavior based on analysis of a video communication by one or more processors, which methods include receiving a user video communication, extracting video analysis data optionally including facial analysis data for the user from the video communication, extracting, by the one or more processors, voice analysis data from the user video communication, generating an outcome prediction score based on the video analysis data and voice analysis data that predicts a likelihood that a user will take an action leading to an outcome.

Claims

1. A video analytics system adapted to predict user behavior based on analysis of a video communication, which comprises:

a node comprising a processor and a non-transitory computer readable medium operably coupled thereto, the non-transitory computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, where the plurality of instructions comprises;

instructions that, when executed, receive a video communication from a user, wherein the video communication comprises an audio component and a video component;

instructions that, when executed, analyze the video component to provide timecoded video behavioral data from the user and identify an attire of the user;

instructions that, when executed, generate an avatar that includes one or more elements of the user'"'"'s attire, wherein the avatar depicts a customer service agent wearing the one or more elements of the user'"'"'s attire;

instructions that, when executed, generate a responsive communication from the avatar to the user;

instructions that, when executed, analyze the audio component to provide timecoded spoken words from the user and from the responsive communication; and

instructions that, when executed, generate an outcome prediction score, based on the time-coded video behavioral data and the time-coded spoken words, that predicts a likelihood that the user will take an action leading to an outcome.

2. The system of claim 1, which further comprises instructions that, when executed, associate the time-coded spoken words with the video behavioral data to determine an emotional state of the user, wherein the generated outcome prediction score is further based on the emotional state of the user.

3. The system of claim 1, which further comprises instructions that, when executed, determine a personality type of the user by applying a linguistic-based algorithm to a text of the spoken words, wherein the generated outcome prediction score is further based on the personality type of the user.

4. The system of claim 1, wherein the user'"'"'s attire is analyzed for elements comprising clothing, jewelry, or accessories.

5. The system of claim 1, wherein the outcome comprises one or more of whether the user will terminate his or her account, whether the user will purchase a product, whether the user is a fraudster, and whether the user will initiate additional subsequent interaction sessions regarding an issue.

6. The system of claim 1, wherein the video component includes a non-verbal, non-textual element comprising one or more of eye movement, facial expressions, gestures, activities, body postures, behaviors, attire, and actions.

7. The system of claim 1, which further comprises instructions that, when executed, determine user value data based on estimated cost of the user'"'"'s attire.

8. The system of claim 1, which further comprises instructions that, when executed, recommend items based on the attire of the user and display one or more of the items in the responsive communication.

9. The system of claim 1, which further comprises instructions that, when executed, determine distress and engagement of the user.

10. The system of claim 9, wherein the outcome prediction score is further based on the user'"'"'s distress and engagement.

11. The system of claim 1, which further comprises instructions that, when executed:

determine, based on the outcome prediction score for a selected outcome of the video communication, a plurality of specific words to include in the responsive communication to the user; and

further develop the responsive communication including the specific words.

12. The system of claim 1, which further comprises instructions that, when executed, generate an avatar that is displayed to the user and provides the responsive communication to the user.

13. A method to predict user behavior based on analysis of a video communication, which comprises:

receiving, by one or more processors, a user video communication;

extracting, by the one or more processors, video analysis data from the user video communication including facial analysis data and identification of an attire of the user;

generating an avatar that includes one or more elements of the user'"'"'s attire, wherein the avatar depicts a customer service agent wearing the one or more elements of the user'"'"'s attire;

generating a responsive communication from the avatar to the user;

extracting, by the one or more processors, voice analysis data from the user in the user video communication and from the responsive communication; and

generating an outcome prediction score, based on the video analysis data including facial analysis data and identification of the attire of the user and based on the voice analysis data, that predicts a likelihood that the user will take an action leading to an outcome.

14. The method of claim 13, which further comprises associating the time-coded spoken words with the video analysis data to determine an emotional state of the user, wherein the generated outcome prediction score is further based on the emotional state of the user.

15. The method of claim 13, which further comprises determining a personality type of the user by applying a linguistic-based algorithm to a text of the spoken words, wherein the generated outcome prediction score is further based on the personality type of the user.

16. The method of claim 13, wherein the user'"'"'s attire is analyzed for elements selected to comprise clothing, jewelry, or accessories.

17. The method of claim 13, wherein the outcome comprises one or more of whether the user will terminate his or her account, whether the user will purchase a product, whether the user is a fraudster, and whether the user will initiate additional subsequent interaction sessions regarding an issue.

18. The method of claim 13, wherein the video analysis data includes a non-verbal, non-textual element comprising one or more of eye movement, facial expressions, gestures, activities, body postures, behaviors, attire, and actions.

19. The method of claim 13, which further comprises determining user value data based on estimated cost of the user'"'"'s attire.

20. The method of claim 13, which further comprises recommending items based on the attire of the user and displaying one or more of the items in the responsive communication.

21. The method of claim 13, which further comprises determining distress and engagement of the user.

22. The method of claim 21, wherein the outcome prediction score is further based on the user'"'"'s distress and engagement.

23. The method of claim 13, which further comprises determining, based on the outcome prediction score for a selected outcome of the video communication, a plurality of specific words to include in the responsive communication to the user;

and further developing the responsive communication including the specific words.

24. A non-transitory machine-readable medium comprising a plurality of instructions which, in response to a computer system, cause the computer system to perform a method which comprises:

receiving a user video communication from a user;

separating an audio component from a video component of the video communication;

analyzing facial expressions of the user in the video component;

identifying an attire of the user in the video component;

generating an avatar that includes one or more elements of the user'"'"'s attire, wherein the avatar depicts a customer service agent wearing the one or more elements of the user'"'"'s attire;

generating a responsive communication from the avatar to the user;

transcribing words spoken of the user in the audio component and from the responsive communication; and

generating an outcome prediction score, based on the analyzed video component and transcribed words, that predicts a likelihood that the user will take an action leading to an outcome.

25. The non-transitory machine-readable medium of claim 24, which further comprises associating the transcribed words with the analyzed facial expressions to determine an emotional state of the user, wherein the generated outcome prediction score is further based on the emotional state of the user.

26. The non-transitory machine-readable medium of claim 24, which further comprises determining a personality type of the user by applying a linguistic-based algorithm to the transcribed words, wherein the generated outcome prediction score is further based on the personality type of the user.

27. The non-transitory machine-readable medium of claim 24, wherein the user'"'"'s attire is analyzed for elements comprising clothing, jewelry, or accessories.

28. The non-transitory machine-readable medium of claim 24, which further comprises determining user value data based on estimated cost of the user'"'"'s attire.

29. The non-transitory machine-readable medium of claim 24, which further comprises recommending items based on the attire of the user and displaying one or more of the items in the responsive communication.

30. The non-transitory machine-readable medium of claim 24, which further comprises determining, based on the outcome prediction score for a selected outcome of the video communication, a plurality of specific words to include in the responsive communication to the user;

and further developing the responsive communication including the specific words.