Establishing a multimodal personality for a multimodal application in dependence upon attributes of user interaction

• US 9,343,064 B2
• Filed: 11/26/2013
• Issued: 05/17/2016
• Est. Priority Date: 09/11/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Cross, Charles W. Jr., Pike, Hilary A.
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US14/090,354
• Publication Number:
  US 20140122091A1
• Time in Patent Office: 903 Days
• Field of Search: 704/275
• US Class Current: 1/1
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

Establishing a multimodal personality for a multimodal application, including evaluating, by the multimodal application, attributes of a user'"'"'s interaction with the multimodal application; selecting, by the multimodal application, a visual demeanor in dependence upon the values of the attributes of the user'"'"'s interaction with the multimodal application; and incorporating, by the multimodal application, the visual demeanor into the multimodal application.

Claims

1. A method of operating a multimodal application to tailor a multimodal personality presented by the multimodal application to a user of the multimodal application, the multimodal application comprising a visual interface and a speech interface, the method comprising:

evaluating speech data of speech of the user, received via the speech interface, to determine at least one attribute of the user'"'"'s speech; and

selecting a selected visual demeanor for the multimodal application based at least in part on the determined at least one attribute of the user'"'"'s speech, wherein the multimodal application can be configured to employ any of a plurality of visual demeanors, wherein the selecting comprises selecting the selected visual demeanor from among the plurality of visual demeanors, each visual demeanor of the plurality of visual demeanors identifying a manner in which to format content for presentation in the visual interface of the multimodal application, each one of the plurality of visual demeanors specifying one or more formatting characteristics to be applied to the content when the one of the plurality of visual demeanors is selected, the one or more formatting characteristics selected from the group consisting of a color in which to present at least some of the content presented in the visual interface, a font in which to present at least some of the content presented in the visual interface, one or more supplemental graphic elements to be added to the content presented in the visual interface, and positions in the visual interface at which to present the content and/or the one or more supplemental graphic elements.

2. The method of claim 1, wherein:

evaluating the speech data to determine the at least one determined attribute comprises determining a gender of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting the selected visual demeanor based on the gender of the user.

3. The method of claim 2, wherein selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies a formatting characteristic that is a particular font and/or type of font to be used in presenting content to users having the gender.

4. The method of claim 1, wherein:

evaluating the speech data to determine the at least one determined attribute comprises determining an accent of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies a formatting characteristic that is a particular graphical element to be presented to users having the accent.

5. The method of claim 1, wherein:

at least some of the plurality of visual demeanors specify formatting characteristics relating to sets of colors to be used in presenting the content in the visual interface; and

selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies one or more formatting characteristics relating to a set of colors based at least in part on the at least one attribute.

6. The method of claim 1, wherein:

each visual demeanor of the plurality of visual demeanors is associated with at least one associated attribute; and

selecting the selected visual demeanor from the plurality of visual demeanors based on the at least one attribute comprises identifying one or more visual demeanors for which at least some of the at least one associated attribute matches the at least one attribute determined from the speech data of speech of the user.

7. At least one non-transitory computer-readable storage medium having encoded thereon executable instructions that, when executed by at least one processor, cause the at least one processor to carry out a method of operating a multimodal application to tailor a multimodal personality presented by the multimodal application to a user of the multimodal application, the multimodal application comprising a visual interface and a speech interface, the method comprising:

evaluating speech data of speech of the user, received via the speech interface, to determine at least one attribute of the user'"'"'s speech; and

selecting a selected visual demeanor for the multimodal application based at least in part on the determined at least one attribute of the user'"'"'s speech, wherein the multimodal application can be configured to employ any of a plurality of visual demeanors, wherein the selecting comprises selecting the selected visual demeanor from among the plurality of visual demeanors, each visual demeanor of the plurality of visual demeanors identifying a manner in which to format content for presentation in the visual interface of the multimodal application, each one of the plurality of visual demeanors specifying one or more formatting characteristics to be applied to the content when the one of the plurality of visual demeanors is selected, the one or more formatting characteristics selected from the group consisting of a color in which to present at least some of the content presented in the visual interface, a font in which to present at least some of the content presented in the visual interface, one or more supplemental graphic elements to be added to the content presented in the visual interface, and positions in the visual interface at which to present the content and/or the one or more supplemental graphic elements.

8. The at least one non-transitory computer-readable storage medium of claim 7, wherein:

evaluating the speech data to determine the at least one determined attribute comprises determining a gender of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting the selected visual demeanor based on the gender of the user.

9. The at least one non-transitory computer-readable storage medium of claim 8, wherein selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies a formatting characteristic that is a particular font and/or type of font to be used in presenting content to users having the gender.

10. The at least one non-transitory computer-readable storage medium of claim 7, wherein:

evaluating the speech data to determine the at least one determined attribute comprises determining an accent of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies a formatting characteristic that is a particular graphical element to be presented to users having the accent.

11. The at least one non-transitory computer-readable storage medium of claim 7, wherein:

at least some of the plurality of visual demeanors specify formatting characteristics relating to sets of colors to be used in presenting the content in the visual interface; and

selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies one or more formatting characteristics relating to a set of colors based at least in part on the at least one attribute.

12. The at least one non-transitory computer-readable storage medium of claim 7, wherein:

the plurality of visual demeanors identify manners in which to present same content via the visual interface; and

the method further comprises;

presenting the same content to the user in the selected visual demeanor via the visual interface;

evaluating second speech data of speech of a second user, received via the speech interface, to determine at least one second attribute of the second user'"'"'s speech;

selecting a second selected visual demeanor for the multimodal application based at least in part on the at least one second attribute of the second user'"'"'s speech; and

presenting the same content to the second user in the second selected visual demeanor via the visual interface.

13. An apparatus comprising:

at least one processor; and

at least one non-transitory computer-readable storage medium having encoded thereon executable instructions that, when executed by at least one processor, cause the at least one processor to carry out a method of operating a multimodal application to tailor a multimodal personality presented by the multimodal application to a user of the multimodal application, the multimodal application comprising a visual interface and a speech interface, the method comprising;

evaluating speech data of speech of the user, received via the speech interface, to determine at least one attribute of the user'"'"'s speech; and

selecting a selected visual demeanor for the multimodal application based at least in part on the determined at least one attribute of the user'"'"'s speech, wherein the multimodal application can be configured to employ any of a plurality of visual demeanors, wherein the selecting comprises selecting the selected visual demeanor from among the plurality of visual demeanors, each visual demeanor of the plurality of visual demeanors identifying a manner in which to format content for presentation in the visual interface of the multimodal application, each one of the plurality of visual demeanors specifying one or more formatting characteristics to be applied to the content when the one of the plurality of visual demeanors is selected, the one or more formatting characteristics selected from the group consisting of a color in which to present at least some of the content presented in the visual interface, a font in which to present at least some of the content presented in the visual interface, one or more supplemental graphic elements to be added to the content presented in the visual interface, and positions in the visual interface at which to present the content and/or the one or more supplemental graphic elements.

14. The apparatus of claim 13, wherein:

evaluating the speech data to determine the at least one determined attribute comprises determining a gender of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting the selected visual demeanor based on the gender of the user.

15. The apparatus of claim 14, wherein selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies a formatting characteristic that is a particular font and/or type of font to be used in presenting content to users having the gender.

16. The apparatus of claim 13, wherein:

evaluating the speech data to determine the at least one determined attribute comprises determining an accent of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies a formatting characteristic that is a particular graphical element to be presented to users having the accent.

17. The apparatus of claim 13, wherein:

at least some of the plurality of visual demeanors specify formatting characteristics relating to sets of colors to be used in presenting the content in the visual interface; and

selecting the selected visual demeanor comprises selecting one of the plurality of visual demeanors that specifies one or more formatting characteristics relating to a set of colors based at least in part on the at least one attribute.

18. The apparatus of claim 13, wherein:

evaluating the speech data to determine the at least one attribute comprises determining an age or age range of the user based at least in part on the speech data; and

selecting the selected visual demeanor comprises selecting a visual demeanor that specifies a set of colors to be used in the visual interface for users having the age or age range.

19. The apparatus of claim 13, wherein:

each visual demeanor of the plurality of visual demeanors is associated with at least one associated attribute; and

selecting the selected visual demeanor from the plurality of visual demeanors based on the at least one attribute comprises identifying one or more visual demeanors for which at least some of the at least one associated attribute matches the at least one attribute determined from the speech data of speech of the user.

20. The apparatus of claim 13, wherein the at least one attribute of the user'"'"'s speech is unrelated to words used by the user in the speech.