Method and system for adjusting the voice prompt of an interactive system based upon the user's state

• US 7,881,934 B2
• Filed: 09/12/2003
• Issued: 02/01/2011
• Est. Priority Date: 09/12/2003
• Status: Active Grant

• Current Assignee: Toyota Infotechnology Center Co., Ltd. (Toyota Motor Corporation)
• Original Assignee: Toyota Infotechnology Center Co., Ltd. (Toyota Motor Corporation)
• Inventors: Reaves, Benjamin K., Endo, Norikazu
• Primary Examiner(s): Wozniak; James S
• Assistant Examiner(s): Serrou; Abdelali
• Application Number:
  US10/661,152
• Publication Number:
  US 20050060158A1
• Time in Patent Office: 2,699 Days
• Field of Search: 704/275, 704/270, 704/270.1, 704/251, 704/257, 704/258
• US Class Current: 704/251
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

The voice prompt of an interactive system is adjusted based upon a state of a user. An utterance of the user is received, and the state of the user is determined based upon signal processing of the utterance of the user. Once the state of the user is determined, the voice prompt is adjusted by adjusting at least one of a tone of voice of the voice prompt, a content of the voice prompt, a prosody of the voice prompt, and a gender of the voice prompt based upon the determined state of the user.

Claims

1. A method of adjusting a voice prompt of a system based upon a state of a user of the system, the method comprising:

receiving an utterance of the user;

obtaining utterance parameters from the utterance, the utterance parameters indicating the state of the user;

determining the state of the user based upon an utterance parameter vector; and

adjusting the voice prompt by adjusting at least one of a tone of voice of the voice prompt, a content of the voice prompt, a prosody of the voice prompt, and a gender of the voice prompt based upon the determined state of the user, wherein obtaining utterance parameters comprises;

partitioning the utterance into segments;

assigning one of a plurality of classifications to each segment, each classification corresponding to at least one of a plurality of states of the user; and

determining a total number of one of the plurality of classifications divided by a total number of segments of the utterance for each of the plurality of states of the user to create the utterance parameter vector.

2. The method of claim 1, wherein converting the utterance parameter vector to an indication comprises applying a function to the utterance parameter vector to generate one of a scalar, a vector of fuzzy classes, and an index representing the state of the user.

3. The method of claim 2, wherein the indication is the scalar and determining the state of the user based upon the indication comprises determining that the user is in a first state if the scalar is greater than a predetermined threshold and that the user is in a second state if the scalar is not greater than the predetermined threshold.

4. The method of claim 1, wherein adjusting the voice prompt comprises adjusting the tone of the voice prompt to use a tone that is consistent with the determined state of the user.

5. The method of claim 1, wherein adjusting the voice prompt comprises adjusting the content of the voice prompt to use content that is consistent with the determined state of the user.

6. The method of claim 1, wherein adjusting the voice prompt comprises adjusting the gender of the voice prompt to use a gender that is consistent with the determined state of the user.

7. The method of claim 1, wherein adjusting the voice prompt comprises adjusting the prosody of the voice prompt to use prosody that is consistent with the determined state of the user.

8. The method of claim 7, wherein adjusting the prosody of the voice prompt comprises pausing the voice prompt.

9. The method of claim 1, wherein the system is an on-board computer used in an automobile or a navigation system used in an automobile.

10. The method of claim 9, further comprising receiving information on a driving condition from the on-board computer or the navigation system and determining the state of the user based upon the information on driving condition.

11. The method of claim 1, further comprising adjusting a graphical character display corresponding to the voice prompt based upon the determined state of the user.

12. A method of adjusting a voice prompt of a system based upon a state of a user of the system, the method comprising:

receiving an utterance of the user;

obtaining utterance parameters from the utterance, the utterance parameters indicating the state of the user;

generating an utterance parameter vector based upon the obtained utterance parameters;

converting the utterance parameter vector to an indication representing the state of the user;

determining the state of the user based upon the indication; and

adjusting the voice prompt based upon the determined state of the user, wherein obtaining utterance parameters comprises;

partitioning the utterance into segments; and

assigning one of a plurality of classifications to each segment, each classification corresponding to at least one of a plurality of states of the user, and wherein generating the utterance parameter vector comprises;

determining the number of segments for each classification; and

dividing the number of segments for each classification by a total number of segments in the utterance.

13. The method of claim 12, wherein converting the utterance parameter vector to the indication comprises applying a function to the utterance parameter vector to generate one of a scalar, a vector of fuzzy classes, and an index representing the state of the user.

14. The method of claim 13, wherein the indication is the scalar and determining the state of the user based upon the scalar comprises determining that the user is in a first state if the scalar is greater than a predetermined threshold and that the user is in a second state if the scalar is not greater than the predetermined threshold.

15. The method of claim 12, wherein adjusting the voice prompt comprises adjusting the tone of the voice prompt to use a tone that is consistent with the determined state of the user.

16. The method of claim 12, wherein adjusting the voice prompt comprises adjusting the content of the voice prompt to use content that is consistent with the determined state of the user.

17. The method of claim 12, wherein adjusting the voice prompt comprises adjusting the gender of the voice prompt to use a gender that is consistent with the determined state of the user.

18. The method of claim 12, wherein adjusting the voice prompt comprises adjusting the prosody of the voice prompt to use prosody that is consistent with the determined state of the user.

19. The method of claim 18, wherein adjusting the prosody of the voice prompt comprises pausing the voice prompt.

20. The method of claim 12, wherein the system is an on-board computer used in an automobile or a navigation system used in an automobile.

21. The method of claim 20, further comprising receiving information on a driving condition from the on-board computer or the navigation system and determining the state of the user based upon the information on driving condition.

22. The method of claim 12, further comprising adjusting a graphical character display corresponding to the voice prompt based upon the determined state of the user.

23. A system adjusting a voice prompt based upon a state of a user of the system, the system comprising:

a signal processing module for obtaining utterance parameters from utterance received from the user, the utterance parameters indicating the state of the user;

an utterance parameter vector generation module for generating an utterance parameter vector based upon the obtained utterance parameters;

a user state determination module for converting the utterance parameter vector to an indication representing the state of the user and determining the state of the user based upon the indication; and

a speech waveform storage module for selecting an audio waveform for the voice prompt based upon the determined state of the user, wherein the signal processing module obtains the utterance parameters by;

partitioning the utterance into segments; and

assigning one of a plurality of classifications to each segment, each classification corresponding to at least one of a plurality of states of the user, wherein the utterance parameter vector generation module generates the utterance parameter vector by;

determining the number of segments for each classification; and

dividing the number of segments for each classification by a total number of segments in the utterance.

24. The system of claim 23, wherein the user state determination module converts the utterance parameter vector to the indication by applying a function to the utterance parameter vector to generate one of a scalar, a vector of fuzzy classes, and an index representing the state of the user.

25. The system of claim 24, wherein the indication is the scalar and the user state determination module determines that the user is in a first state if the scalar is greater than a predetermined threshold and that the user is in a second state if the scalar is not greater than the predetermined threshold.

26. The system of claim 23, wherein the speech waveform storage module selects the audio waveform of the voice prompt having a tone that is consistent with the determined state of the user.

27. The system of claim 23, wherein the speech waveform storage module selects the audio waveform of the voice prompt having content that is consistent with the determined state of the user.

28. The system of claim 23, wherein the speech waveform storage module selects the audio waveform of the voice prompt of a gender that is consistent with the determined state of the user.

29. The system of claim 23, wherein the speech waveform storage module selects the audio waveform of the voice prompt having prosody that is consistent with the determined state of the user.

30. The system of claim 23, wherein the system is an on-board computer used in an automobile or a navigation system used in an automobile.

31. The system of claim 30, wherein the user state determination module receives information on a driving condition from the on-board computer or the navigation system and determines the state of the user based upon the information on driving condition.

32. The system of claim 23, further comprising a display device for displaying a graphical character corresponding to the voice prompt to the user, the displayed graphical character being adjusted based upon the determined state of the user.

33. A system adjusting a voice prompt based upon a state of a user of the system, the system comprising:

a signal processing module for obtaining utterance parameters from utterance received from the user, the utterance parameters indicating the state of the user;

an utterance parameter vector generation module for generating an utterance parameter vector based upon the obtained utterance parameters;

a user state determination module for converting the utterance parameter vector to an indication representing the state of the user and determining the state of the user based upon the indication; and

a speech synthesizer module for generating an audio waveform of the voice prompt based upon the determined state of the user, wherein the signal processing module obtains the utterance parameters by;

partitioning the utterance into segments; and

assigning one of a plurality of classifications to each segment, each classification corresponding to at least one of a plurality of states of the user, wherein the utterance parameter vector generation module generates the utterance parameter vector by;

determining the number of segments for each classification; and

dividing the number of segments for each classification by a total number of segments in the utterance.

34. The system of claim 33, wherein the user state determination module converts the utterance parameter vector to the indication by applying a function to the utterance parameter vector to generate one of a scalar, a vector of fuzzy classes, and an index representing the state of the user.

35. The system of claim 34, wherein the indication is the scalar and the user state determination module determines that the user is in a first state if the scalar is greater than a predetermined threshold and that the user is in a second state if the scalar is not greater than the predetermined threshold.

36. The system of claim 33, wherein the speech synthesizer module generates the audio waveform of the voice prompt to have a tone that is consistent with the determined state of the user.

37. The system of claim 33, wherein the speech synthesizer module generates the audio waveform of the voice prompt based upon content that is consistent with the determined state of the user.

38. The system of claim 33, wherein the speech synthesizer module generates the audio waveform of the voice prompt to be of a gender that is consistent with the determined state of the user.

39. The system of claim 33, wherein the speech synthesizer module generates the audio waveform of the voice prompt having prosody that is consistent with the determined state of the user.

40. The system of claim 33, further comprising a speech storage module storing speech and outputting speech that is consistent with the determined state of the user to the speech synthesizer module.

41. The system of claim 33, wherein the system is an on-board computer used in an automobile or a navigation system used in an automobile.

42. The system of claim 41, wherein the user state determination module receives information on a driving condition from the on-board computer or the navigation system and determines the state of the user based upon the information on driving condition.

43. The system of claim 33, further comprising a display device for displaying a graphical character corresponding to the voice prompt to the user, the displayed graphical character being adjusted based upon the determined state of the user.

44. A system adjusting a voice prompt based upon a state of a user of the system, the system comprising:

means for obtaining utterance parameters from utterance received from the user, the utterance parameters indicating the state of the user;

means for determining the state of the user based upon the utterance parameters; and

means for adjusting the voice prompt by adjusting at least one of a tone of voice of the voice prompt, a content of the voice prompt, a prosody of the voice prompt, and a gender of the voice prompt based upon the determined state of the user, wherein the means for obtaining utterance parameters comprises;

means for partitioning the utterance into segments; and

means for assigning one of a plurality of classifications to each segment, each classification corresponding to at least one of a plurality of states of the user, wherein the means for determining the state of the user based upon the utterance comprises;

means for generating an utterance parameter vector based upon the utterance parameter by (1) determining the number of segments for each classification, and (2) dividing the number of segments for each classification by a total number of segments in the utterance.

45. The system of claim 44, wherein the means for determining the state of the user based upon the utterance comprises:

means for generating an utterance parameter vector based upon the utterance parameters;

means for converting the utterance parameter vector to an indication representing the state of the user; and

means for determining the state of the user based upon the indication.

46. The system of claim 45, wherein the means for converting the utterance parameter vector to the indication applies a function to the utterance parameter vector to generate one of a scalar, a vector of fuzzy classes, and an index representing the state of the user.

47. The system of claim 46, wherein the indication is the scalar and the means for determining the state of the user based upon the indication determines that the user is in a first state if the scalar is greater than a predetermined threshold and that the user is in a second state if the scalar is not greater than the predetermined threshold.

48. The system of claim 44, wherein the means for adjusting the voice prompt adjusts the tone of the voice prompt to use a tone that is consistent with the determined state of the user.

49. The system of claim 44, wherein the means for adjusting the voice prompt adjusts the content of the voice prompt to use content that is consistent with the determined state of the user.

50. The system of claim 44, wherein the means for adjusting the voice prompt adjusts the gender of the voice prompt to use a gender that is consistent with the determined state of the user.

51. The system of claim 44, wherein the means for adjusting the voice prompt adjusts the prosody of the voice prompt to use prosody that is consistent with the determined state of the user.

52. The system of claim 44, wherein the system is an on-board computer used in an automobile or a navigation system used in an automobile.