Speech dialogue system for facilitating improved human-computer interaction

• US 5,357,596 A
• Filed: 11/18/1992
• Issued: 10/18/1994
• Est. Priority Date: 11/18/1991
• Status: Expired due to Term

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation), Toshiba Software Engineering Corporation (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation), Toshiba Software Engineering Corporation (Toshiba Corporation)
• Inventors: Tsuboi, Hiroyuki, Nagata, Yoshifumi, Shinchi, Hideaki, Sadamoto, Yoichi, Yamashita, Yasuki, Seto, Shigenobu, Hashimoto, Hideki, Takebayashi, Yoichi
• Primary Examiner(s): MacDonald, Allen R.
• Assistant Examiner(s): Doerrler, Michelle
• Application Number:
  US07/978,521
• Time in Patent Office: 699 Days
• Field of Search: 381/41-45, 395/2.79, 395/2.84, 395/2.4, 395/2.6
• US Class Current: 704/275
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

A speech dialogue system capable of realizing natural and smooth dialogue between the system and a human user, and easy maneuverability of the system. In the system, a semantic content of input speech from a user is understood and a semantic content determination of a response output is made according to the understood semantic content of the input speech. Then, a speech response and a visual response according to the determined response output are generated and outputted to the user. The dialogue between the system and the user is managed by controlling transitions between user states during which the input speech is to be entered and system states during which the system response is to be outputted. The understanding of a semantic content of input speech from a user is made by detecting keywords in the input speech, with the keywords to be detected in the input speech limited in advance, according to a state of a dialogue between the user and the system.

Claims

1. A speech dialogue system, comprising:

speech understanding means for understanding a semantic content of an input speech from a user;

dialogue management means for making a semantic content determination of a response output according to said semantic content of said input speech understood by said speech understanding means;

response generation means for generating a speech response and a visual response according to said semantic content of said response output determined by said dialogue management means; and

output means for outputting to said user said speech response and said visual response generated by said response generation means.

2. The speech dialogue system of claim 1, wherein said response generation means generates said visual response which includes an image of a human character delivering said speech response, text data of said speech response, and a content visualizing image of a content of said speech response.

3. The speech dialogue system of claim 1, wherein said output means outputs said speech response and said visual response by controlling at least one of an output order, an output timing, and a visual response output position.

4. The speech dialogue system of claim 1, further comprising user state detection means for detecting a physical state of said user, wherein said physical state of said user detected by said user state detection means is taken into a account by said dialogue management means in making said semantic content determination of said response output.

5. The speech dialogue system of claim 1, wherein said response generation means generates said visual response which includes an image of a human character delivering said speech response, wherein said image incorporates movement and facial expression of said human character.

6. The speech dialogue system of claim 5, wherein said response generation means generates said speech response incorporating a speech characteristic corresponding to said movement and said facial expression of said human character.

7. The speech dialogue system of claim 6, wherein said speech characteristic of said speech response includes at least one of an emotional expression and an intonation.

8. The speech dialogue system of claim 1, wherein said speech understanding means supplies a plurality of candidates for said semantic content of said input speech, and said dialogue management means makes said semantic content determination of said response output by evaluating said plurality of candidates in accordance with a dialogue history.

9. The speech dialogue system of claim 1, wherein said output means also outputs a visual indication for informing said user as to whether said speech dialogue system is ready to receive said input speech.

10. The speech dialogue system of claim 1, wherein said response generation means generates said visual response including a content visualizing image formed by pictures of objects mentioned in said speech response and a numerical figure indicating a quantity of each of said objects.

11. The speech dialogue system of claim 1, wherein said response generation means generates said speech response for making a confirmation of said semantic content of said input speech, while generating said visual response on the basis of a past history of a dialogue between said user and said speech dialogue system.

12. The speech dialogue system of claim 1, wherein said response generation means generates said visual response which include text data of said speech response and graphic images other than said text data, and said response generation means generates said speech response and said text data for making a confirmation of said semantic content of said input speech while generating said graphic images on the basis of a past history of a dialogue between said user and said speech dialogue system.

13. The speech dialogue system of claim 1, wherein said response generation means generates said speech response for making a confirmation of said semantic content of said input speech, said speech response being changed from a full speech response to a simplified Speech response according to a length of said speech response for making said confirmation.

14. The speech dialogue system of claim 13, wherein said length of said speech response for making said confirmation is determined from information items to be confirmed by said confirmation.

15. The speech dialogue system of claim 14, wherein said full speech response mentions all of said information items to be confirmed while said simplified speech response does not directly mention said information items to be confirmed.

16. The speech dialogue system of claim 15, wherein said simplified speech response contains a demonstrative pronoun to refer to said visual response.

17. The speech dialogue system of claim 13, wherein said full speech response recites said response output explicitly while said simplified speech response does not recite said response output explicitly.

18. The speech dialogue system of claim 17, wherein said simplified speech response contains a demonstrative pronoun to refer to said visual response.

19. The speech dialogue system of claim 13, wherein said output means outputs said visual response at an earlier timing that a timing for outputting said visual response when said speech response is said full speech response.

20. The speech dialogue system of claim 13, wherein said output means outputs said visual response before said speech response is outputted.

21. A method of speech dialogue between a human user and a speech dialogue system, comprising the steps of:

understanding a semantic content of an input speech from a user;

making a semantic content determination of a response output according to said semantic content of said input speech;

generating a speech response and a visual response according to said response output; and

outputting to said user said speech response and said visual response.

22. The method of claim 21, wherein said generating step generates said visual response which includes an image of a human character delivering said speech response, text data of said speech response, and a content visualizing image of a content of said speech response.

23. The method of claim 21, wherein said outputting step outputs said speech response and said visual response by controlling at least one of an output order, an output timing, and a visual response output position.

24. The method of claim 21, further comprising the step of detecting a physical state of said user, and wherein said making step makes said semantic content determination of said response output by taking said physical state of said user detected at said detecting step into account.

25. The method of claim 21, wherein said generating step generates said visual response which includes an image of a human character delivering said speech response, where said image incorporates movement and facial expression of said human character.

26. The method of claim 25, wherein said generating step generates said speech response incorporating a speech characteristic corresponding to said movement and said facial expression of said human character.

27. The method of claim 26, wherein said speech characteristic of said speech response includes at least one of an emotional expression and an intonation.

28. The method of claim 21, wherein said understanding step obtains a plurality of candidates for said semantic content of Said input speech, and said making step makes said semantic content determination of said response output by evaluating said plurality of candidates according to a dialogue history.

29. The method of claim 21, wherein said outputting step also outputs a visual indication for informing said user as to whether said speech dialogue system is ready to receive said input speech.

30. The method of claim 21, wherein said generating step generates said visual response which includes a content visualizing image formed by pictures of objects mentioned in said speech response and a numerical figure indicating a quantity of each of said objects.

31. The method of claim 21, wherein at said generating step, said speech response for making a confirmation of said semantic content of said input speech is generated, while said visual response reflecting a past history of a dialogue between said user and said speech dialogue system is generated.

32. The method of claim 21, wherein at said generating step, said visual response which includes text data of said speech response and graphic images other than said text data is generated, and said speech response and said text data for making a confirmation of said semantic content of said input speech is generated while graphic images reflecting a past history of a dialogue between said user and said speech dialogue system are generated.

33. The method of claim 21, wherein at said generating step, said speech response for making a confirmation of said semantic content of said input speech is generated, said speech response being changed from a full speech response to a simplified speech response according to a length of said speech response for making said confirmation.

34. The method of claim 33, wherein said length of said speech response for making said confirmation is determined from information items to be confirmed by said confirmation.

35. The method of claim 34, wherein said full speech response mentions all of said information items to be confirmed while said simplified speech response does not directly mention said information items to be confirmed.

36. The method of claim 35, wherein said simplified speech response contains a demonstrative pronoun to refer to said visual response.

37. The method of claim 33, wherein said full speech response recites said response output explicitly while said simplified speech response does not recite said response output explicitly.

38. The method of claim 37, wherein said simplified speech response contains a demonstrative pronoun to refer to said visual response.

39. The method of claim 33, wherein at said outputting step, said visual response is outputted at an earlier timing than a timing for outputting said visual response when said speech response is said full speech response.

40. The method of claim 33, wherein at said outputting step, said visual response is outputted before said speech response is outputted.

41. A speech dialogue system, comprising:

speech understanding means for understanding a semantic content of an input speech from a user;

response output means for outputting a system response according to said semantic content of said input speech understood by said speech understanding means; and

dialogue management means for managing a dialogue between said user and said speech dialogue system by controlling transitions between user states during which said input speech is to be entered into said speech understanding means and system states during which said system response is to be outputted from said response output means.

42. The speech dialogue system of claim 41, wherein said dialogue management means controls said transitions between said user states and said system states according to said semantic content of said input speech understood by said speech understanding means.

43. A speech dialogue system, comprising:

speech understanding means for understanding a semantic content of an input speech from a user by detecting keywords of said input speech;

dialogue management means for limiting said keywords to be detected in said input speech in advance, according to a state of a dialogue between said user and said speech dialogue system; and

response output means for outputting a system response according to said semantic content of said input speech understood by said speech understanding means.

44. The speech dialog system of claim 43, wherein said dialogue management means makes a semantic content determination of said system response to be outputted by said response output means in accordance with said semantic content of said input speech and understood by said speech understanding means, and limits said keywords to be detected in said input speech by said speech understanding means in advance by using said semantic content determination of said system response for previous input speech which is indicative of a current state of said dialogue between said user and said speech dialogue system.

45. The speech dialogue system of claim 43, wherein said dialogue management means also limits syntactic and semantic rules to be used by said speech understanding means in advance, according to a state of a dialogue between said user and said speech dialogue system.