Generating self-support metrics based on paralinguistic information

• US 10,573,311 B1
• Filed: 10/02/2018
• Issued: 02/25/2020
• Est. Priority Date: 10/26/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Indyk, Benjamin, Podgorny, Igor A., Chan, Raymond
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US16/150,197
• Time in Patent Office: 511 Days
• Field of Search: 704 7, 704246, 704247, 704251, 704252
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06Q 40/12 Accounting
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: G10L 25/87 Detection of discrete points within a voice signal

Abstract

Certain aspects of the present disclosure provide techniques for selecting a response to a self-support query. One example method generally includes receiving an audio stream query including spoken content from a user recorded by a mobile device and determining a set of paralinguistic features from the spoken content. The method further includes estimating an emotional state of the user based on the set of paralinguistic features and identifying subject matter of the spoken content in the audio stream query. The method further includes determining two or more query responses corresponding to the subject matter to present to the user and transmitting at least one query response to the mobile device.

Claims

1. A method for selecting a response to a self-support query, the method comprising:

receiving an audio stream query including spoken content from a user recorded by a mobile device;

determining a set of paralinguistic features from the audio stream query;

estimating an emotional state of the user based on the set of paralinguistic features;

identifying subject matter of the spoken content in the audio stream query;

determining two or more query responses corresponding to the subject matter;

determining one query response from the two or more query responses to present to the user based on an uplift model metric generated by an uplift model that includes instructions to provide the one query response related to the identified subject and the estimated emotional state of the user; and

transmitting the one query response to the mobile device.

2. The method of claim 1, wherein the uplift model has been trained using a set of A/B test results corresponding to a plurality of spoken queries referencing the subject matter and using a set of user attributes comprising at least one user attribute corresponding to each of the plurality of spoken queries.

3. The method of claim 2, wherein the set of A/B test results are gathered according to A/B test metrics, wherein the A/B test metrics define:

parameters of an A/B test;

a plurality of user reactions to the A/B test; and

a favorability score corresponding to each user reaction.

4. The method of claim 2, wherein the set of user attributes comprise emotional state attributes.

5. The method of claim 1, wherein identifying the subject matter of the spoken content in the audio stream query comprises:

comparing the spoken content to types of subject matter in a database.

6. The method of claim 1, wherein estimating the emotional state of the user based on the set of paralinguistic features comprises:

comparing the paralinguistic features to attributes in a speech database.

7. The method of claim 1, further comprising:

receiving, from the mobile device, a user reaction to the one query response;

determining a favorability metric associated with the user reaction; and

updating the uplift model based on the emotional state of the user, the user reaction, and the favorability metric.

8. The method of claim 7, further comprising:

upon updating the uplift model, transmitting an additional query response to the mobile device.

9. A computing device, comprising:

a processor; and

a memory including computer readable instructions, which, when executed by the processor, cause the computing device to perform a method for selecting a response to a self-support query, the method comprising;

receiving an audio stream query including spoken content from a user recorded by a mobile device;

determining a set of paralinguistic features from the audio stream query;

estimating an emotional state of the user based on the set of paralinguistic features;

identifying subject matter of the spoken content in the audio stream query;

determining two or more query responses corresponding to the subject matter;

determining one query response from the two or more query responses to present to the user based on an uplift model metric generated by an uplift model that includes instructions to provide the one query response related to the identified subject and the estimated emotional state of the user; and

transmitting the one query response to the mobile device.

10. The computing device of claim 9, wherein the uplift model has been trained using a set of A/B test results corresponding to a plurality of spoken queries referencing the subject matter and using a set of user attributes comprising at least one user attribute corresponding to each of the plurality of spoken queries.

11. The computing device of claim 10, wherein the set of A/B test results are gathered according to A/B test metrics, wherein the A/B test metrics define:

parameters of an A/B test;

a plurality of user reactions to the A/B test; and

a favorability score corresponding to each user reaction.

12. The computing device of claim 10, wherein the set of user attributes comprise emotional state attributes.

13. The computing device of claim 9, wherein identifying the subject matter of the spoken content in the audio stream query comprises:

comparing the spoken content to types of subject matter in a database.

14. The computing device of claim 9, wherein estimating the emotional state of the user based on the set of paralinguistic features comprises:

comparing the paralinguistic features to attributes in a speech database.

15. The computing device of claim 9, the method further comprising:

receiving, from the mobile device, a user reaction to the one query response;

determining a favorability metric associated with the user reaction; and

updating the uplift model based on the emotional state of the user, the user reaction, and the favorability metric.

16. The computing device of claim 15, the method further comprising:

upon updating the uplift model, transmitting an additional query response to the mobile device.

17. A non-transitory computer readable medium comprising instructions, which when executed by a processor of a computing device, cause the computing device to perform a method for selecting a response to a self-support query, the method comprising:

receiving an audio stream query including spoken content from a user recorded by a mobile device;

determining a set of paralinguistic features from the audio stream query;

estimating an emotional state of the user based on the set of paralinguistic features;

identifying subject matter of the spoken content in the audio stream query;

determining two or more query responses corresponding to the subject matter;

determining one query response from the two or more query responses to present to the user based on an uplift model metric generated by an uplift model that includes instructions to provide the one query response related to the identified subject and the estimated emotional state of the user; and

transmitting the one query response to the mobile device.

18. The non-transitory computer readable medium of claim 17, wherein the uplift model has been trained using a set of A/B test results corresponding to a plurality of spoken queries referencing the subject matter and using a set of user attributes comprising at least one user attribute corresponding to each of the plurality of spoken queries, wherein the set of A/B test results are gathered according to A/B test metrics, and wherein the A/B test metrics define:

parameters of an A/B test;

a plurality of user reactions to the A/B test; and

a favorability score corresponding to each user reaction.

19. The non-transitory computer readable medium of claim 17, wherein identifying the subject matter of the spoken content in the audio stream query comprises:

comparing the spoken content to types of subject matter in a database.

20. The non-transitory computer readable medium of claim 17, the method further comprising:

receiving, from the mobile device, a user reaction to the one query response;

determining a favorability metric associated with the user reaction;

updating the uplift model based on the emotional state of the user, the user reaction, and the favorability metric; and

upon updating the uplift model, transmitting an additional query response to the mobile device.