Personalized support routing based on paralinguistic information

• US 10,623,573 B2
• Filed: 09/09/2019
• Issued: 04/14/2020
• Est. Priority Date: 10/27/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Indyk, Benjamin, Podgorny, Igor A., Chan, Raymond
• Primary Examiner(s): Addy, Thjuan K
• Application Number:
  US16/564,462
• Publication Number:
  US 20200007687A1
• Time in Patent Office: 218 Days
• Field of Search: 37926513, 379 8801, 37926511, 37926512, 37926505, 37926501, 37926601
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2203/2038 Call context notifications
• CPC Class Codes: H04M 3/42068 where the identifier is used to access a profile
• CPC Class Codes: H04M 3/5233 Operator skill based call distribution
• CPC Class Codes: H04M 3/5235 Dependent on call type or called number [DNIS]

Abstract

Embodiments presented herein provide techniques for inferring the current emotional state of a user based on paralinguistic features derived from audio input from that user. If the emotional state meets triggering conditions, the system provides the user with a prompt which allows the user to connect with a support agent. If the user accepts, the system selects a support agent for the user based on the predicted emotional state and on attributes of the support agent found in an agent profile. The system can also determine a priority level for the user based on the score and based on a profile of the user and determine where to place the user in a queue for the support agent.

Claims

1. A computer-implemented method comprising:

retrieving a set of audio recordings of each interaction between a support agent and each user in a set of users;

extracting, from each audio recording, a set of paralinguistic features associated with each user;

generating, based on the set of paralinguistic features, a score for each user that represents an emotional state of the user during the interaction with the support agent;

determining an implicit customer-satisfaction level for each user interacting with the support agent based on the score;

obtaining a set of explicit customer-satisfaction levels from the set of users corresponding to the interaction with the support agent;

determining a customer-satisfaction attribute of the support agent based on each implicit customer-satisfaction level and the set of explicit customer-satisfaction levels; and

generating a support agent profile for the support agent including the customer-satisfaction attribute.

2. The computer-implemented method of claim 1, further comprises:

providing a survey to each user interacting with the support agent.

3. The computer-implemented method of claim 2, wherein the explicit customer-satisfaction level is based on the survey completed by the user after interacting with the support agent.

4. The computer-implemented method of claim 1, wherein the customer-satisfaction attribute is an average of each customer-satisfaction level.

5. The computer-implemented method of claim 1, wherein determining the implicit customer-satisfaction level is based on a function.

6. The computer-implemented method of claim 1, wherein the score is generated by a predictive model trained to predict the emotional state of the user based on a set of training audio data and corresponding labels representing the score associated with each training audio data.

7. The computer-implemented method of claim 1, wherein extracting the set of paralinguistic features includes identifying each portion of the audio recording the user is speaking.

8. A system, comprising:

a processor; and

a memory storing one or more applications, which, when executed on the processor, perform an operation, the operation comprising;

retrieving a set of audio recordings of each interaction between a support agent and each user in a set of users;

extracting, from each audio recording, a set of paralinguistic features associated with each user;

generating, based on the set of paralinguistic features, a score for each user that represents an emotional state of the user during the interaction with the support agent;

determining an implicit customer-satisfaction level for each user interacting with the support agent based on the score;

obtaining a set of explicit customer-satisfaction levels from the set of users corresponding to the interaction with the support agent;

determining a customer-satisfaction attribute of the support agent based on each implicit customer-satisfaction level and the set of explicit customer-satisfaction levels; and

generating a support agent profile for the support agent including the customer-satisfaction attribute.

9. The system of claim 8, further comprises:

providing a survey to each user interacting with the support agent.

10. The system of claim 9, wherein the explicit customer-satisfaction level is based on the survey completed by the user after interacting with the support agent.

11. The system of claim 8, wherein the customer-satisfaction attribute is an average of each customer-satisfaction level.

12. The system of claim 8, wherein determining the implicit customer-satisfaction level is based on a function.

13. The system of claim 8, wherein the score is generated by a predictive model trained to predict the emotional state of the user based on a set of training audio data and corresponding labels representing the score associated with each training audio data.

14. The system of claim 8, wherein extracting the set of paralinguistic features includes identifying each portion of the audio recording the user is speaking.

15. A non-transitory computer-readable storage medium containing instructions that, when executed by one or more processors, perform an operation, the operation comprising:

retrieving a set of audio recordings of each interaction between a support agent and each user in a set of users;

extracting, from each audio recording, a set of paralinguistic features associated with each user;

generating, based on the set of paralinguistic features, a score for each user that represents an emotional state of the user during the interaction with the support agent;

determining an implicit customer-satisfaction level for each user interacting with the support agent based on the score;

obtaining a set of explicit customer-satisfaction levels from the set of users corresponding to the interaction with the support agent;

determining a customer-satisfaction attribute of the support agent based on each implicit customer-satisfaction level and the set of explicit customer-satisfaction levels; and

generating a support agent profile for the support agent including the customer-satisfaction attribute.

16. The non-transitory computer-readable storage medium of claim 15, further comprises:

providing a survey to each user interacting with the support agent.

17. The non-transitory computer-readable storage medium of claim 16, wherein the explicit customer-satisfaction level is based on the survey completed by the user after interacting with the support agent.

18. The non-transitory computer-readable storage medium of claim 15, wherein the customer-satisfaction attribute is an average of each customer-satisfaction level.

19. The non-transitory computer-readable storage medium of claim 15, wherein determining the implicit customer-satisfaction level is based on a function.

20. The non-transitory computer-readable storage medium of claim 15, wherein the score is generated by a predictive model trained to predict the emotional state of the user based on a set of training audio data and corresponding labels representing the score associated with each training audio data.