Method and apparatus for tailoring the output of an intelligent automated assistant to a user

• US 9,213,558 B2
• Filed: 09/01/2010
• Issued: 12/15/2015
• Est. Priority Date: 09/02/2009
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Tur, Gokhan, Franco, Horacio E., Shriberg, Elizabeth, Myers, Gregory K., Mark, William S., Winarsky, Norman D., Stolcke, Andreas, Peintner, Bart, Wolverton, Michael J., Ferrer, Luciana, Graciarena, Martin, Bratt, Harry, Yorke-Smith, Neil
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Bharadwaj, Kalpana
• Application Number:
  US13/378,512
• Publication Number:
  US 20120166365A1
• Time in Patent Office: 1,931 Days
• Field of Search: 706/11
• US Class Current: 1/1
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

The present invention relates to a method and apparatus for tailoring the output of an intelligent automated assistant. One embodiment of a method for conducting an interaction with a human user includes collecting data about the user using a multimodal set of sensors positioned in a vicinity of the user, making a set of inferences about the user in accordance with the data, and tailoring an output to be delivered to the user in accordance with the set of inferences.

Claims

1. A method for conducting an interaction with a human user, the method comprising:

collecting data about the user using a multimodal set of sensors positioned in a vicinity of the user;

making a set of inferences about the user in accordance with the data; and

tailoring an output to be delivered to the user in accordance with the set of inferences wherein making the set of inferences comprises extracting at least one feature from the data; and

classifying the at least one feature in accordance with at least one model that defines a potential characteristic of the user.

2. The method of claim 1, wherein the set of inferences further relates to one or more personal characteristics of the user.

3. The method of claim 2, wherein the one or more personal characteristics include an age of the user.

4. The method of claim 2, wherein the one or more personal characteristics include a gender of the user.

5. The method of claim 2, wherein the one or more personal characteristics include a socioeconomic group to which the user belongs.

6. The method of claim 1, wherein the plurality of features include at least one of:

a lexical content of an utterance made by the user or a linguistic content of an utterance made by the user.

7. The method of claim 1, wherein the plurality of features include at least one of:

one or more pauses within an utterance made by the user, one or more increments in a duration of phones uttered by the user relative to a pre-computed average, a latency of the user to produce a response to a prompt, a probability distribution of unit durations, or timing information related to one or more user interruptions to a previous output.

8. The method of claim 1, wherein the plurality of features include at least one of:

a fundamental frequency range within an utterance made by the user, a fundamental frequency slope along one or more words, a probability distribution of a slope, or a probability distribution of a plurality of fundamental frequency values.

9. The method of claim 1, wherein the plurality of features include at least one of:

a range of energy excursions within an utterance made by the user, a slope of energy within an utterance made by the user, a probability distribution of normalized energy, or a probability distribution of energy slopes.

10. The method of claim 1, wherein the plurality of features include at least one of:

a color of at least a portion of a face of the user, a shape of at least a portion of a face of the user, a texture of at least a portion of a face of the user, an orientation of at least a portion of a face of the user, or a movement of at least a portion of a face of the user.

11. The method of claim 1, wherein the plurality of features include at least one of:

whether the user is looking at a display on which the output is to be presented, a percentage of time spent by the user looking at a display on which the output is to be presented, a part of a display on which the output is to be presented on which the user is focused, how close a focus of the user is to a desired part of a display on which the output is to be presented, or a percentage of time spent by the user looking at a desired part of a display on which the output is to be presented.

12. The method of claim 1, wherein the plurality of features include at least one of:

a shape of an area below a face of the user, a color of an area below a face of the user, or a texture of an area below a face of the user.

13. The method of claim 1, wherein the plurality of features include at least one of:

a pose of a portion of a body of the user as a function of time or a motion of a portion of a body of the user as a function of time.

14. The method of claim 1, wherein the plurality of features include at least one of:

a shape of footwear worn by the user, a color of footwear worn by the user, or a texture of footwear worn by the user.

15. The method of claim 1, wherein the classifying is performed using a statistical classifier.

16. The method of claim 1, wherein the classifying is performed using a training-based classifier.

17. A non-transitory computer readable medium containing an executable program for conducting an interaction with a human user, where the program performs steps comprising:

collecting data about the user using a multimodal set of sensors positioned in a vicinity of the user;

making a set of inferences about the user in accordance with the data; and

tailoring an output to be delivered to the user in accordance with the set of inferences wherein making the set of inferences comprises extracting at least one feature from the data; and

classifying the at least one feature in accordance with at least one model that defines a potential characteristic of the user.

18. A system for conducting an interaction with a human user, the system comprising:

a plurality of multimodal sensors positioned in a vicinity of the user for collecting data about the user;

a plurality of classifiers for making a set of inferences about the user in accordance with the data; and

an output selection module for tailoring an output to be delivered to the user in accordance with the set of inferences wherein making the set of inferences comprises extracting at least one feature from the data; and

classifying the at least one feature in accordance with at least one model that defines a potential characteristic of the user.