Generating self-support metrics based on paralinguistic information

• US 10,147,424 B1
• Filed: 10/26/2016
• Issued: 12/04/2018
• Est. Priority Date: 10/26/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Indyk, Benjamin, Podgorny, Igor A., Chan, Raymond
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US15/334,825
• Time in Patent Office: 769 Days
• Field of Search: 704 7, 704246, 704247, 704251, 704252
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

The present disclosure includes techniques for selecting a response to an audio stream query. In one embodiment, an application server receives an audio stream query including content spoken by a user interacting with a voice-user interface. The application server determines a set of paralinguistic features from the audio stream query, and estimates at least a first attribute of the user based on the set of paralinguistic features. The application server identifies subject matter corresponding to the spoken content in the audio stream query, and determines two or more query responses corresponding to the identified subject matter. The application server then selects one of the query responses to present to the user based, at least in part, on the attribute of the user estimated from the set of paralinguistic features.

Claims

1. A computer-implemented method for selecting a response to an audio stream query, the method comprising:

receiving, over a network, a self-support query including spoken content from a user recorded by a remote mobile device;

determining a set of paralinguistic features from the spoken content;

estimating an emotional state of the user based on the set of paralinguistic features;

identifying text strings representing subject matter based on the spoken content in the audio stream query;

determining two or more query responses corresponding to the text strings representing the subject matter;

selecting, from a query response database, one of the query responses to present to the user based, at least in part, on the emotional state of the user and an uplift model used to select query responses for users with the emotional state;

receiving, over the network from the remote mobile device, a user reaction to the query response;

determining a favorability metric associated with the user reaction;

updating the uplift model based on the attribute of the user, the user reaction and the favorability metric; and

based on updating the uplift model, transmitting an additional query response to the remote mobile device, wherein the additional query response is selected based on the uplift model for users with the emotional state of the user.

2. The method of claim 1, wherein the uplift model is trained using a set of A/B test results corresponding to a plurality of spoken queries referencing the identified subject matter and using a set of user attributes comprising at least one user attribute corresponding to each of the plurality of spoken queries.

3. The method of claim 2, wherein the A/B test results are gathered according to A/B test metrics, wherein the A/B test metrics define:

parameters of an A/B test;

a plurality of user reactions to the A/B test; and

a favorability score corresponding to each user reaction.

4. The method of claim 2, wherein the set of user attributes comprise emotional state attributes.

5. The method of claim 1, wherein the attribute of the user estimated from the set of paralinguistic features comprises an emotional state of the user.

6. The method of claim 1, wherein estimating the attribute of the user based on the set of paralinguistic features comprises comparing the paralinguistic features to attributes in the speech database.

7. The method of claim 1, wherein identifying subject matter corresponding to the spoken content in the audio stream query comprises comparing the spoken content to types of subject matter in a database.

8. A non-transitory computer-readable storage medium containing instructions that, when executed by one or more processors, perform an operation for selecting a response to an audio stream query, the operation comprising:

receiving, over a network, a self-support query including spoken content from a user recorded by a remote mobile device;

determining a set of paralinguistic features from the spoken content;

estimating an emotional state of the user based on the set of paralinguistic features;

identifying text strings representing subject matter based on the spoken content in the audio stream query;

determining two or more query responses corresponding to the text strings representing the subject matter;

selecting, from a query response database, one of the query responses to present to the user based, at least in part, on the emotional state of the user and an uplift model used to select query responses for users with the emotional state;

receiving, over the network from the remote mobile device, a user reaction to the query response;

determining a favorability metric associated with the user reaction;

updating the uplift model based on the attribute of the user, the user reaction and the favorability metric; and

based on updating the uplift model, transmitting an additional query response to the remote mobile device, wherein the additional query response is selected based on the uplift model for users with the emotional state of the user.

9. The computer-readable storage medium of claim 8, wherein the uplift model is trained using a set of A/B test results corresponding to a plurality of spoken queries referencing the identified subject matter and using a set of user attributes comprising at least one user attribute corresponding to each of the plurality of spoken queries.

10. The computer-readable storage medium of claim 9, wherein the A/B test results are gathered according to A/B test metrics, wherein the A/B test metrics define:

parameters of an A/B test;

a plurality of user reactions to the A/B test; and

a favorability score corresponding to each user reaction.

11. The computer-readable storage medium of claim 9, wherein the set of user attributes comprise emotional state attributes.

12. The computer-readable storage medium of claim 8, wherein the attribute of the user estimated from the set of paralinguistic features comprises an emotional state of the user.

13. The computer-readable storage medium of claim 8, wherein estimating the attribute of the user based on the set of paralinguistic features comprises comparing the paralinguistic features to attributes in the speech database.

14. The computer-readable storage medium of claim 8, wherein identifying subject matter corresponding to the spoken content in the audio stream query comprises comparing the spoken content to types of subject matter in a database.

15. A system, comprising:

one or more processors; and

memory storing one or more applications, which, when executed on the one or more processors perform an operation for selecting a response to an audio stream query, the operation comprising;

receiving, over a network, a self-support query including spoken content from a user recorded by a remote mobile device;

determining a set of paralinguistic features from the spoken content;

estimating an emotional state of the user based on the set of paralinguistic features;

identifying text strings representing subject matter based on the spoken content in the audio stream query;

determining two or more query responses corresponding to the text strings representing the subject matter;

selecting, from a query response database, one of the query responses to present to the user based, at least in part, on the emotional state of the user and an uplift model used to select query responses for users with the emotional state;

receiving, over the network from the remote mobile device, a user reaction to the query response;

determining a favorability metric associated with the user reaction;

updating the uplift model based on the attribute of the user, the user reaction and the favorability metric; and

based on updating the uplift model, transmitting an additional query response to the remote mobile device, wherein the additional query response is selected based on the uplift model for users with the emotional state of the user.

16. The system of claim 15, wherein the uplift model is trained using a set of A/B test results corresponding to a plurality of spoken queries referencing the identified subject matter and using a set of user attributes comprising at least one user attribute corresponding to each of the plurality of spoken queries.

17. The system of claim 15, wherein the attribute of the user estimated from the set of paralinguistic features comprises an emotional state of the user.

18. The system of claim 16, wherein the A/B test results are gathered according to A/B test metrics, wherein the A/B test metrics define:

parameters of an A/B test;

a plurality of user reactions to the A/B test; and

a favorability score corresponding to each user reaction.

19. The system of claim 15, wherein estimating the attribute of the user based on the set of paralinguistic features comprises comparing the paralinguistic features to attributes in the speech database.

20. The system of claim 15, wherein identifying subject matter corresponding to the spoken content in the audio stream query comprises comparing the spoken content to types of subject matter in a database.