Automatic generation of playlists from conversations

• US 10,324,979 B2
• Filed: 10/25/2017
• Issued: 06/18/2019
• Est. Priority Date: 06/01/2015
• Status: Active Grant

• Current Assignee: Zoominfo Converse LLC
• Original Assignee: AffectLayer, Inc. (ZoomInfo Technologies, Inc.)
• Inventors: Raanani, Roy, Levy, Russell, Breakstone, Micha Yochanan
• Primary Examiner(s): Addy, Thjuan K
• Application Number:
  US15/793,691
• Publication Number:
  US 20180046710A1
• Time in Patent Office: 601 Days
• Field of Search: 704235, 37926507, 37926505, 37926506, 3792661, 37926503
• CPC Class Codes: G06F 16/639 using playlists
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/02 Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2203/301 : Management of recordings
• CPC Class Codes: H04M 2203/305 : Recording playback features, e.g. increased speed
• CPC Class Codes: H04M 2203/357 : Autocues for dialog assistance
• CPC Class Codes: H04M 2203/556 : Statistical analysis and interpretation
• CPC Class Codes: H04M 3/42221 : Conversation recording systems at the subscriber's set H04M1/656
• CPC Class Codes: H04M 3/5175 : Call or contact centers supervision arrangements
• CPC Class Codes: H04M 3/5232 : Call distribution algorithms
• CPC Class Codes: H04N 7/141 : between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085

Abstract

A moment identification system automatically generates a playlist of conversations having a specified moment. A moment can be occurrence of a specific event or a specific characteristic in a conversation, or any event that is of specific interest for an application for which the playlist is being generated. For example, a moment can include laughter, fast-talking, objections, response to questions, a discussion on a particular topic such as budget, behavior of a speaker, intent to buy, etc., in a conversation. The moment identification system analyzes each of the conversations to determine if one or more features of a conversation correspond to a specified moment, and includes those of the conversations in the playlist having one or more features that correspond to the specified moment. The playlist may include a portion of a conversation that has the specified moment rather than the entire conversation.

Claims

1. A computer-implemented method, comprising:

retrieving multiple recordings of conversations between multiple participants, the multiple participants including multiple customers and multiple representatives, wherein each conversation includes at least one of the multiple customers and at least one of the multiple representatives;

extracting, from each of the multiple recordings, multiple features of a corresponding conversation, wherein the multiple features indicate characteristics of any of (a) a customer of the multiple customers in the corresponding conversation, (b) a participant of the multiple participants in the corresponding conversation, (c) the corresponding conversation;

identifying multiple moments of the conversations, wherein a specified moment of the multiple moments indicates an occurrence of a specified event in a corresponding conversation, the specified event characterized by one or more of the multiple features; and

generating a playlist for the specified moment, the playlist including recordings of those of the conversations having the specified moment.

2. The computer-implemented method of claim 1, wherein generating the playlist includes:

receiving, from a requesting user, a request to generate the playlist for the specified moment,

identifying those of the multiple recordings that have the specified moment to generate multiple identified recordings,

for each of the multiple identified recordings, determining a snippet of the corresponding identified recording that have the specified moment, and

generating the playlist by including the snippet from each of the identified recordings.

3. The computer-implemented method of claim 2, wherein the snippet includes a portion of a conversation corresponding to the identified recording that has (a) the specified moment, (b) a first duration of the conversation prior to the specified moment, and (c) a second duration of the conversation after the specified moment.

4. The computer-implemented method of claim 1, wherein extracting the multiple features includes:

generating features that include a transcription, vocabulary and a language model of the conversations as a first output.

5. The computer-implemented method of claim 4, wherein extracting the multiple features includes:

generating, using the first output, features that include semantic information from the conversations.

6. The computer-implemented method of claim 1, wherein extracting the multiple features includes:

generating a set of low-level features that indicate information associated with a voice signal in the multiple recordings, and a set of high-level features that include personality traits and personal attributes of the multiple participants and emotion data that indicates emotions of the multiple participants.

7. The computer-implemented method of claim 1, wherein extracting the multiple features includes generating features that include data regarding conversation flow.

8. The computer-implemented method of claim 1, wherein extracting the multiple features includes generating features related a representative-customer pair in a conversation of the conversations.

9. The computer-implemented method of claim 1, wherein extracting the multiple features includes:

generating a speaker engagement metric that includes information regarding a degree of engagement of a specified customer of the multiple customers in a conversation of the multiple conversations.

10. The computer-implemented method of claim 1, wherein extracting the multiple features includes extracting a visual feature associated with a conversation of the conversations.

11. The computer-implemented method of claim 1, wherein identifying the moments of the corresponding conversation includes:

identifying, for the specified moment and based on the multiple features of the corresponding conversation, a portion of the corresponding conversation at which the specified event occurred, the specified event indicative of a customer-related feature, a representative-related feature, or a conversation-related feature.

12. The computer-implemented method of claim 1, wherein identifying the moments of the corresponding conversation includes:

identifying an “

interesting”

moment in the corresponding conversation, the interesting moment identified based on user-defined criteria, usage pattern of a recording of the corresponding conversation, or anomaly in the corresponding conversation.

13. The computer-implemented method of claim 12, wherein identifying the interesting moment based on the anomaly includes:

identifying the anomaly based on a change in one of the multiple features of the corresponding conversation, an occurrence of a specified feature of the multiple features, or a frequency of occurrence of the specified feature exceeding a specified threshold.

14. The computer-implemented method of claim 12, wherein identifying the interesting moment based on the usage pattern includes:

identifying the usage pattern based on a number of times a specified portion of a recording of the corresponding conversation has been played, whether the specified portion is associated with any comment, or a speed or volume level at which the specified portion is played compared to other portions.

15. The computer-implemented method of claim 1, wherein generating the playlist includes generating a context-aware playlist, the context-aware playlist being a playlist generated for the specified moment and for those of the multiple conversations that satisfy a specified context.

16. The computer-implemented method of claim 15, wherein generating the context-aware playlist includes generating the context-aware playlist by selecting those of the multiple conversations that include a specified participant of the multiple participants.

17. The computer-implemented method of claim 15, wherein generating the context-aware playlist includes generating the context-aware playlist by selecting those of the multiple conversations that were carried out in a specified geographical location.

18. The computer-implemented method of claim 15, wherein generating the context-aware playlist includes generating the context-aware playlist by selecting those of the multiple conversations that were carried out at a specified time.

19. The computer-implemented method of claim 1, wherein extracting the multiple features includes extracting the multiple features using any of an artificial intelligence, a machine learning technique, or natural language processing technique.

20. The computer-implemented method of claim 1, wherein at least one of the multiple recordings includes a recording of a video call between one of the customers and one of the representatives.

21. The computer-implemented method of claim 1, wherein at least one of the multiple recordings includes an online meeting between one of the customers and one of the representatives.

22. The computer-implemented method of claim 1, wherein at least one of the multiple recordings includes a recording of a virtual reality-based conversation between one of the customers and one of the representatives.

23. The computer-implemented method of claim 1, wherein at least one of the multiple recordings includes a recording of an augmented reality-based conversation between one of the customers and one of the representatives.

24. The computer-implemented method of claim 1, wherein at least one of the multiple recordings includes an e-mail conversation between one of the customers and one of the representatives.

25. A non-transitory computer-readable storage medium storing computer-readable instructions, comprising:

instructions for receiving a request to generate a playlist having a set of call recordings having a specified moment, each call recording from the set of call recordings being a recording of a conversation between at least two participants, wherein one of the at least two participants is a specified representative of multiple representatives of an organization and the other of the at least two participants is a customer, wherein the specified moment indicates an occurrence of a specified event in the conversation;

instructions for extracting, from each of multiple recordings stored in a storage system, multiple features of a corresponding conversation, wherein the multiple features indicate characteristics of any of (a) a customer of the multiple customers in the corresponding conversation, (b) a representative of the multiple representatives in the corresponding conversation, (c) the corresponding conversation;

instructions for identifying those of the conversations having the specified moment as a set of conversations, wherein the specified moment is identified based on one or more of the multiple features associated with the conversations that are representative of the specified event; and

instructions for generating the playlist having the set of call recordings corresponding to the set of conversations.

26. The non-transitory computer-readable storage medium of claim 25, wherein the instructions for extracting the multiple features include:

instructions for generating at least one of aural features or video features associated with the conversation.

27. The non-transitory computer-readable storage medium of claim 25, wherein the specified moment is based on a specified video feature associated with the conversations, wherein the instructions for generating the playlist include:

instructions for generating the playlist having those of the multiple recordings which are associated with the specified video feature.

28. The non-transitory computer-readable storage medium of claim 27, wherein the instructions for generating the playlist based on the specified video feature include:

instructions for generating the playlist having those of the multiple recordings in which one of the at least two participants made a specified facial expression.

29. The non-transitory computer-readable storage medium of claim 25, wherein the specified moment is based on a specified aural feature associated with the conversations, wherein the instructions for generating the playlist include:

instructions for generating the playlist having those of the multiple recordings which are associated with the specified aural feature.

30. The non-transitory computer-readable storage medium of claim 29, wherein the instructions for generating the playlist based on the specified aural feature include:

instructions for generating the playlist having those of the multiple recordings in which one of the at least two participants uttered a specified word, discussed a specified topic, asked a specific question, or answered the specific question.

31. The non-transitory computer-readable storage medium of claim 25 further comprising:

instructions for generating multiple playlists for multiple moments; and

instructions for tagging each of the multiple playlists with information regarding the corresponding moment for which the playlist is generated.

32. The non-transitory computer-readable storage medium of claim 31 further comprising:

instructions for receiving a specified request for generating a specified playlist for a particular moment;

instructions for determining, based on the tagging, whether any of the multiple playlists is generated for the particular moment; and

instructions for returning, in response to the specified request, one of the multiple playlists that is tagged with the particular moment as the specified playlist.

33. The non-transitory computer-readable storage medium of claim 25, wherein the instructions for generating the playlist include:

instructions for identifying a portion of a conversation in a recording of the multiple recordings that has (a) the specified moment, (b) a first duration of the conversation prior to the specified moment, and (c) a second duration of the conversation after the specified moment, and

instructions for including a recording of the portion of the conversation in the playlist.

34. The non-transitory computer-readable storage medium of claim 25, wherein the instructions for identifying those of the conversations having the specified moment include:

instructions for identifying an “

interesting”

moment in a conversation of the multiple conversations, the interesting moment identified based on user-defined criteria, usage pattern of a recording of the conversation, or anomaly in the conversation.

35. The non-transitory computer-readable storage medium of claim 34, wherein the instructions for identifying the interesting moment based on the anomaly include:

instructions for identifying the anomaly based on a change in one of the multiple features of the conversation, an occurrence of a specified feature of the multiple features, or a frequency of occurrence of the specified feature exceeding a specified threshold.

36. The non-transitory computer-readable storage medium of claim 34, wherein the instructions for identifying the interesting moment based on the usage pattern include:

instructions for identifying the usage pattern based on a number of times a specified portion of the recording has been played, whether the specified portion is associated with any comment, or a speed or volume level at which the specified portion is played compared to other portions.

37. A system for automatic generation of a playlist of call recordings having a specified moment, the system comprising:

a first component that is configured to retrieve multiple recordings of conversations between multiple participants, the multiple participants including multiple customers and multiple representatives, wherein each conversation includes at least one of the multiple customers and at least one of the multiple representatives;

a second component that is configured to extract multiple features from the multiple recordings, wherein the multiple features includes characteristics of the multiple customers, the multiple representatives, and the conversations;

a third component that is configured to analyze the multiple recordings to identify the specified moment of the conversations, wherein the specified moment indicates an occurrence of a specified event in a conversation of the conversations, the specified event characterized by one or more of the multiple features; and

a fourth component that is configured to generate a playlist for the specified moment, the playlist including a recording of (a) a first conversation of the multiple conversations and (b) a second conversation of the multiple conversations having the specified moment.

38. The system of claim 37, wherein the fourth component is configured to generate the playlist by including a first snippet of the recording of the first conversation and a second snippet of the recording of the second conversation.

39. The system of claim 37, wherein the fourth component is configured to generate the playlist by generating a context-aware playlist, the context-aware playlist being a playlist generated for the specified moment and for those of the multiple conversations that satisfy a specified context.

40. The system of claim 39, wherein the fourth component is configured to:

select those of the multiple conversations that include a specified participant of the multiple participants to generate selected conversations, and

include recordings of the selected conversations in the context-aware playlist.