Personalized support routing based on paralinguistic information

• US 10,412,223 B2
• Filed: 11/01/2018
• Issued: 09/10/2019
• Est. Priority Date: 10/27/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Indyk, Benjamin, Podgorny, Igor A., Chan, Raymond
• Primary Examiner(s): Addy, Thjuan K
• Application Number:
  US16/177,510
• Publication Number:
  US 20190075204A1
• Time in Patent Office: 313 Days
• Field of Search: 37926513, 379 8801, 37926511, 37926512, 37926505, 37926501, 37926601
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/14 using statistical models, e.g. Hidden Markov Models [HMMs] G10L15/18 takes precedence
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2203/2038 Call context notifications
• CPC Class Codes: H04M 3/42068 where the identifier is used to access a profile
• CPC Class Codes: H04M 3/5233 Operator skill based call distribution
• CPC Class Codes: H04M 3/5235 Dependent on call type or called number [DNIS]

Abstract

Embodiments presented herein provide techniques for inferring the current emotional state of a user based on paralinguistic features derived from audio input from that user. If the emotional state meets triggering conditions, the system provides the user with a prompt which allows the user to connect with a support agent. If the user accepts, the system selects a support agent for the user based on the predicted emotional state and on attributes of the support agent found in an agent profile. The system can also determine a priority level for the user based on the score and based on a profile of the user and determine where to place the user in a queue for the support agent.

Claims

1. A computer-implemented method comprising:

receiving an audio recording of an interaction between a first user and a support agent;

extracting a first set of paralinguistic features from audio of the first user during a first time interval in the audio recording, wherein the first set of paralinguistic features characterize acoustic aspects of the audio of the first user during the first time interval distinct from verbal content of the audio during the first time interval;

predicting, from the first set of paralinguistic features, a first emotional state of the first user during the first time interval using a predictive model;

extracting a second set of paralinguistic features from audio of the first user during a second time interval in the audio recording, wherein the second set of paralinguistic features characterize acoustic aspects of the audio of the first user during the second time interval distinct from verbal content of the audio during the second time interval;

predicting, from the second set of paralinguistic features, a second emotional state of the first user during the second time interval using the predictive model;

determining one or more attributes of the support agent based on the first emotional state and the second emotional state; and

generating a profile of the support agent including the one or more attributes.

2. The computer-implemented method of claim 1, wherein determining the one or more attributes comprises:

generating, using the predictive model, a first score for the first emotional state;

generating, using the predictive model, a second score for the second emotional state; and

determining a first customer-satisfaction level based on a difference between the first score and the second score.

3. The computer-implemented method of claim 2, wherein the one or more attributes comprise a customer-satisfaction attribute.

4. The computer-implemented method of claim 3, further comprising selecting the support agent to communicate with a second user if the customer-satisfaction attribute satisfies a predetermined threshold.

5. The computer-implemented method of claim 3, wherein the customer-satisfaction attribute is determined based on the first customer-satisfaction level and at least a second customer-satisfaction level associated with another audio recording of an interaction between a second user and the support agent.

6. The computer-implemented method of claim 5, wherein the customer-satisfaction attribute is a median of at least the first and second customer-satisfaction levels.

7. The computer-implemented method of claim 1, wherein the first time interval does not overlap with the second time interval.

8. A system, comprising:

a processor; and

a memory storing one or more applications, which, when executed on the processor, perform an operation, the operation comprising;

receiving an audio recording of an interaction between a first user and a support agent;

extracting a first set of paralinguistic features from audio of the first user during a first time interval in the audio recording, wherein the first set of paralinguistic features characterize acoustic aspects of the audio of the first user during the first time interval distinct from verbal content of the audio during the first time interval;

predicting, from the first set of paralinguistic features, a first emotional state of the first user during the first time interval using a predictive model;

extracting a second set of paralinguistic features from audio of the first user during a second time interval in the audio recording, wherein the second set of paralinguistic features characterize acoustic aspects of the audio during the second time interval distinct from verbal content of the audio of the first user during the second time interval;

predicting, from the second set of paralinguistic features, a second emotional state of the first user during the second time interval using the predictive model;

determining one or more attributes of the support agent based on the first emotional state and the second emotional state; and

generating a profile of the support agent including the one or more attributes.

9. The system of claim 8, wherein determining the one or more attributes comprises:

generating, using the predictive model, a first score for the first emotional state;

generating, using the predictive model, a second score for the second emotional state; and

determining a first customer-satisfaction level based on a difference between the first score and the second score.

10. The system of claim 9, wherein the one or more attributes comprise a customer-satisfaction attribute.

11. The system of claim 10, the operation further comprising selecting the support agent to communicate with a second user if the customer-satisfaction attribute satisfies a predetermined threshold.

12. The system of claim 10, wherein the customer-satisfaction attribute is determined based on the first customer-satisfaction level and at least a second customer-satisfaction level associated with another audio recording of an interaction between a second user and the support agent.

13. The system of claim 12, wherein the customer-satisfaction attribute is a median of at least the first and second customer-satisfaction levels.

14. The system of claim 13, wherein the first time interval does not overlap with the second time interval.

15. A non-transitory computer-readable storage medium containing instructions that, when executed by one or more processors, perform an operation, the operation comprising:

receiving an audio recording of an interaction between a first user and a support agent;

extracting a first set of paralinguistic features from audio of the first user during a first time interval in the audio recording, wherein the first set of paralinguistic features characterize acoustic aspects of the audio of the first user during the first time interval distinct from verbal content of the audio during the first time interval;

predicting, from the first set of paralinguistic features, a first emotional state of the first user during the first time interval using a predictive model;

extracting a second set of paralinguistic features from audio of the first user during a second time interval in the audio recording, wherein the second set of paralinguistic features characterize acoustic aspects of the audio of the first user during the second time interval distinct from verbal content of the audio during the second time interval;

predicting, from the second set of paralinguistic features, a second emotional state of the first user during the second time interval using the predictive model;

determining one or more attributes of the support agent based on the first emotional state and the second emotional state; and

generating a profile of the support agent including the one or more attributes.

16. The non-transitory computer-readable storage medium of claim 15, wherein determining the one or more attributes comprises:

generating, using the predictive model, a first score for the first emotional state;

generating, using the predictive model, a second score for the second emotional state; and

determining a first customer-satisfaction level based on a difference between the first score and the second score.

17. The non-transitory computer-readable storage medium of claim 16, wherein the one or more attributes comprise a customer-satisfaction attribute.

18. The non-transitory computer-readable storage medium of claim 17, the operation further comprising selecting the support agent to communicate with a second user if the customer-satisfaction attribute satisfies a predetermined threshold.

19. The non-transitory computer-readable storage medium of claim 17, wherein the customer-satisfaction attribute is determined based on the first customer-satisfaction level and at least a second customer-satisfaction level associated with another audio recording of an interaction between a second user and the support agent.

20. The non-transitory computer-readable storage medium of claim 19, wherein the customer-satisfaction attribute is a median of at least the first and second customer-satisfaction levels.