Method and apparatus for tailoring the output of an intelligent automated assistant to a user

• US 9,501,743 B2
• Filed: 12/02/2015
• Issued: 11/22/2016
• Est. Priority Date: 09/02/2009
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Tur, Gokhan, Franco, Horacio E., Shriberg, Elizabeth, Myers, Gregory K., Mark, William S., Winarsky, Norman D., Stolcke, Andreas, Peintner, Bart, Wolverton, Michael J., Ferrer, Luciana, Graciarena, Martin, Yorke-Smith, Neil, Bratt, Harry
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Bharadwaj, Kalpana
• Application Number:
  US14/957,286
• Publication Number:
  US 20160086090A1
• Time in Patent Office: 356 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

The present invention relates to a method and apparatus for tailoring the output of an intelligent automated assistant. One embodiment of a method for conducting an interaction with a human user includes collecting data about the user using a multimodal set of sensors positioned in a vicinity of the user, making a set of inferences about the user in accordance with the data, and tailoring an output to be delivered to the user in accordance with the set of inferences.

Claims

1. A method for conducting an interaction with a user, the method comprising:

collecting data about the user using at least one audio sensor positioned in a vicinity of the user;

extracting feature data from the collected data using a plurality of feature extractors, wherein the feature data includes at least one feature of the collected data;

combining the feature data from the plurality of feature extractors to produce combined features;

modeling ones of the combined features as joint features; and

classifying at least one of the joint features by at least one classifier using at least one model that defines an affective state of the user in accordance with the collected data; and

tailoring an output to be delivered to the user in accordance with the affective state.

2. The method of claim 1, wherein the affective state relates to one or more personal characteristics of the user.

3. The method of claim 2, wherein the one or more personal characteristics include at least one an age, gender or socioeconomic group of the user.

4. The method of claim 1, wherein the at least one feature includes at least one of:

a lexical content of an utterance made by the user or a linguistic content of an utterance made by the user;

one or more pauses within an utterance made by the user;

one or more increments in a duration of phones uttered by the user relative to a pre-computed average;

a latency of the user to produce a response to a prompt;

a probability distribution of unit durations;

timing information related to one or more user interruptions to a previous output a fundamental frequency range within an utterance made by the user;

a fundamental frequency slope along one or more words;

a probability distribution of a slope;

a probability distribution of a plurality of fundamental frequency values;

a range of energy excursions within an utterance made by the user,

a slope of energy within an utterance made by the user;

a probability distribution of normalized energy; and

a probability distribution of energy slopes.

5. The method of claim 1, wherein the classifying is performed using at least one of a statistical classifier, a training-based classifier.

16. The method of claim 1, further comprising conditioning the feature data based on an intrinsic value for the received feature data.

6. A non-transitory computer readable medium containing an executable program for conducting an interaction with a user, where the program performs steps comprising:

collecting data about the user using at least one audio sensor positioned in a vicinity of the user;

extracting feature data from the collected data using a plurality of feature extractors, wherein the feature data includes at least one feature of the collected data;

combining the feature data from the plurality of feature extractors to produce combined features;

modeling ones of the combined features as joint features;

classifying at least one of the joint features by at least one classifier using at least one model that defines an affective state of the user in accordance with the collected data; and

tailoring an output to be delivered to the user in accordance with the affective state.

12. The non-transitory computer readable medium of claim 6, wherein the affective state relates to one or more personal characteristics of the user.

14. The non-transitory computer readable medium of claim 6, wherein the at least one feature includes at least one of:

a lexical content of an utterance made by the user or a linguistic content of an utterance made by the user;

one or more pauses within an utterance made by the user;

one or more increments in a duration of phones uttered by the user relative to a pre-computed average;

a latency of the user to produce a response to a prompt;

a probability distribution of unit durations;

timing information related to one or more user interruptions to a previous output a fundamental frequency range within an utterance made by the user;

a fundamental frequency slope along one or more words;

a probability distribution of a slope;

a probability distribution of a plurality of fundamental frequency values;

a range of energy excursions within an utterance made by the user,

a slope of energy within an utterance made by the user;

a probability distribution of normalized energy; and

a probability distribution of energy slopes.

15. The non-transitory computer readable medium of claim 6, wherein the classifying is performed using at least one of a statistical classifier, a training-based classifier.

17. The non-transitory computer readable medium of claim 6, wherein the steps further comprise conditioning the feature data based on an intrinsic value for the received feature data.

7. A system for conducting an interaction with a user, the system comprising:

at least one audio sensor positioned in a vicinity of the user for collecting data about the user;

a plurality of feature extractors for receiving the collected data and extracting feature data including at least one feature from the collected data;

a feature combination module for combining the feature data received from the plurality of feature extractors to produce combined features and modeling ones of the combined features as joint features;

at least one classifier for classifying at least one of the joint features using at least one model that defines an affective state of the user in accordance with the collected data; and

an output selection module for tailoring an output to be delivered to the user in accordance with the affective state.

8. The system of claim 7, wherein the feature data includes at least one of:

a lexical content of an utterance made by the user or a linguistic content of an utterance made by the user;

at least one of one or more pauses within an utterance made by the user;

one or more increments in a duration of phones uttered by the user relative to a pre-computed average;

a latency of the user to produce a response to a prompt;

a probability distribution of unit durations;

timing information related to one or more user interruptions to a previous output;

a fundamental frequency range within an utterance made by the user;

a fundamental frequency slope along one or more words;

a probability distribution of a slope;

a probability distribution of a plurality of fundamental frequency values;

a range of energy excursions within an utterance made by the user;

a slope of energy within an utterance made by the user; and

a probability distribution of normalized energy, or a probability distribution of energy slopes.

9. The system of claim 7, wherein the at least one classifier includes a statistical classifier or a training-based classifier.

10. The system of claim 7, wherein the classifying of at least one of the joint features defines at least one of an age, gender or socioeconomic characteristic of the user.

11. The system of claim 7, wherein the plurality of feature extractors includes at least one of an automatic speech recognition and natural language understanding processor, a duration feature extractor, a pitch feature extractor, an energy feature extractor, and a cepstral feature extractor.

13. The non-transitory computer readable medium of claim 11, wherein the one or more personal characteristics include at least one an age, gender or socioeconomic group of the user.

18. The system of claim 7, wherein the feature combination module also conditions the feature data based on an intrinsic value for the received feature data.