Method of and apparatus for managing dialog between user and agent

• US 7,734,468 B2
• Filed: 12/11/2003
• Issued: 06/08/2010
• Est. Priority Date: 12/11/2002
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Hong, Young-jin, Lee, Jae-won, Park, Joon-ah
• Primary Examiner(s): Wozniak; James S
• Application Number:
  US10/732,235
• Publication Number:
  US 20040122673A1
• Time in Patent Office: 2,371 Days
• Field of Search: 704/251, 704/257, 704/270, 704/275, 382116-118, 379/88.01
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

A method of rapidly and precisely managing a dialog turn between a user and an agent by using speech information, facial expression information, and delay time information includes generating first dialog turn information using dialog information analyzed from a speech uttered by the user, generating second dialog turn information using facial expression information analyzed from a face image of the user, and determining a final dialog turn using the first and second dialog turn information, information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user.

Claims

1. A method of managing a dialog turn between a user and a spoken dialog system, the method comprising:

storing weights assigned to varying dialog turn information based on statistical information in a weight database;

generating first dialog turn information using dialog information analyzed from a speech uttered by the user;

generating second dialog turn information using facial expression information analyzed from a face image of the user;

synthesizing the first dialog turn information and the second dialog turn information and outputting the synthesized dialog turn information when the first dialog turn information is different than the second dialog turn information, and outputting one selected from the first dialog turn information and the second dialog turn information when the first dialog turn information is identical to the second dialog turn information;

searching the weight database for weights corresponding to the synthesized dialog turn information, assigning a positive weight to each of the first and second dialog turn information based on the synthesized dialog turn information and selecting whichever one of the first and second dialog turn information has a greater positive weight; and

determining a final dialog turn using the selected one dialog turn information, information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user, for controlling a dialog between the user and the spoken dialog system.

2. The method of claim 1, wherein the generating the first dialog turn information comprises:

recognizing the user speech and analyzing the user speech into morphemes;

extracting sentence pattern information, modal information, and discourse marker information using the morphemes; and

generating the first dialog turn information using the sentence pattern information, the modal information, and the discourse marker information.

3. The method of claim 2, wherein the generating the first dialog turn information further comprises, when at least one dialog turn information corresponding to the sentence pattern information, the modal information, and the discourse marker information exist, generating the first dialog turn information with reference to a recent dialog history.

4. The method of claim 1, wherein the generating the second dialog turn information comprises:

obtaining information on face, eye, eyebrow, and eyelid patterns from the user face image;

comparing the obtained information with standard pattern information to generate information on movements of the user; and

generating the second dialog turn information using the movement information.

5. The method of claim 1, wherein the determining a final dialog turn comprises:

determining from the status information on the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determining the final dialog turn depending on the information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determining the final dialog turn depending on the information on the user no-answer time or the selected one dialog turn information.

6. The method of claim 5, wherein the determining the final dialog turn depending on the information on whether the user speech is input and the selected one dialog turn information comprises:

determining whether the user speech is input;

if it is determined that the user speech has been input, controlling the spoken dialog system so as to stop speaking; and

if it is determined that the user speech has not been input, controlling the spoken dialog system so as to one of stop, keep, and commence speaking determined based on the selected one dialog turn information.

7. The method of claim 6, wherein the controlling the spoken dialog system so as to stop, keep, or commence speaking based on the selected one dialog turn information comprises:

if it is determined that the user speech has not been input, a residual amount of an answer being presently spoken by the spoken dialog system is checked, and

if the residual amount of the answer being presently spoken is to be finished within a predetermined period of time and the final dialog turn is determined as the turn-take, the spoken dialog is controlled to continue speaking, otherwise the spoken dialog is controlled to stop speaking.

8. The method of claim 5, wherein the determining the final dialog turn depending on the information on the user no-answer time or the selected one dialog turn information comprises:

determining whether the user no-answer time exceeds a predetermined period of time;

if it is determined that the user no-answer time exceeds the predetermined period of time, controlling the spoken dialog system so as to commence speaking; and

if it is determined that the user no-answer time does not exceed the predetermined period of time, controlling the spoken dialog system to one of stop, keep, and commence speaking determined based on the selected one dialog turn information.

9. A computer-readable storage medium encoded with computer readable code to control a computer to implement the method of claim 1.

10. An apparatus for managing a dialog turn between a user and a spoken dialog system including a computer having computing device-executable instructions, the apparatus comprising:

a dialog analyzer, controlled by the computer, that generates first dialog turn information using dialog information analyzed from a speech uttered by the user;

a facial expression analyzer, controlled by the computer, that generates second dialog turn information using facial expression information analyzed from a face image of the user;

a weight database, controlled by the computer, that stores weights assigned to varying dialog turn information based on statistical information;

a synthesizer, controlled by the computer, that synthesizes the first dialog turn information and the second dialog turn information, the synthesizer outputting the synthesized dialog turn information to a weight searcher when the first dialog turn information is different than the second dialog turn information and outputting one selected from the first dialog turn information and the second dialog turn information to a dialog controller when the first dialog turn information is identical to the second dialog turn information;

the weight searcher searching the weight database for weights corresponding to the synthesized dialog turn information output from the synthesizer, assigning a positive weight to each of the first and second dialog turn information based on the synthesized dialog turn information and selecting whichever one of the first and second dialog turn information has a greater positive weight; and

the dialog controller determining a final dialog turn using the selected one dialog turn information, information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user, for controlling a dialog between the user and the spoken dialog system.

11. The apparatus of claim 10, wherein the dialog analyzer comprises:

a morpheme analyzer that analyzes the user speech into morphemes;

a keyword extractor that extracts sentence pattern information, modal information, and discourse marker information using the morphemes;

a sentence pattern database that stores information on speech acts and dialog turns resulting from the sentence pattern information, the modal information, and the discourse marker information; and

a sentence pattern searcher that searches the sentence pattern database using the sentence pattern information, the modal information, and the discourse marker information to generate the first dialog turn information.

12. The apparatus of claim 11, wherein the dialog analyzer further comprises:

a dialog history database that stores information on speech acts of a recent dialog sentence; and

a candidate dialog turn selector that selects the first dialog turn information with reference to the dialog history database when the sentence pattern searcher searches out at least one dialog turn information from the sentence pattern database.

13. The apparatus of claim 10, wherein the facial expression analyzer comprises:

an image normalizer that normalizes the user face image so as to have a predetermined size and color;

a face pattern database that stores data on feature points for recognition of face, eye, eyebrow, and eyelid patterns;

a face pattern searcher that searches the face pattern database to obtain information on locations of face, eye, eyebrow, and eyelid patterns of the normalized face image;

a movement information generator that compares the obtained position information with information on locations of standard patterns to generate movement information;

a facial expression pattern database that stores information on facial expression patterns corresponding to the determined movement information and dialog turns of current speakers; and

a facial expression pattern searcher that searches the facial expression pattern database using the movement information to generate the second dialog turn information.

14. The apparatus of claim 10, wherein the dialog controller:

determines from the status information on the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determines the final dialog turn based on the information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determines the final dialog turn based the user no-answer time and the selected one dialog turn information.

15. A spoken dialog system including a computer having computing device-executable instructions, the system comprising:

a speech input unit that amplifies a user speech input via a microphone to a predetermined level;

a speech recognizer, controlled by the computer, that recognizes the speech signal provided by the speech input unit to output a character string signal;

an image input unit that inputs a user face image;

a timer that counts a user no-answer time for a predetermined period of time;

a weight database, controlled by the computer, that stores weights assigned to varying dialog turn information based on statistical information;

a dialog manager, controlled by the computer, that generates first dialog turn information using dialog information analyzed from the character string signal and generates second dialog turn information using facial expression information analyzed from the user face image;

a synthesizer, controlled by the computer, that synthesizes the first dialog turn information and the second dialog turn information, the synthesizer outputting the synthesized dialog turn information to a weight searcher when the first dialog turn information is different than the second dialog turn information and outputting one selected from the first dialog turn information and the second dialog turn information to a dialog controller when the first dialog turn information is identical to the second dialog turn information;

the weight searcher searching the weight database for weights corresponding to the synthesized dialog turn information output from the synthesizer, assigning a positive weight to each of the first and second dialog turn information based on the synthesized dialog turn information and selecting whichever one of the first and second dialog turn information has a greater positive weight;

the dialog controller determining a final dialog turn using the selected one dialog turn information; and

an answer generator, controlled by the computer, that generates an answer corresponding to the character string signal with reference to a dialog model database according to a control operation of the dialog-controller.

16. A method of managing a dialog turn between a user and a spoken dialog system, the method comprising:

storing weights assigned to varying dialog turn information based on statistical information in a weight database;

generating first dialog turn information using dialog information analyzed from a speech uttered by the user;

generating second dialog turn information using body expression information analyzed from a detected body expression of the user;

synthesizing the first dialog turn information and the second dialog turn information and outputting the synthesized dialog turn information when the first dialog turn information is different than the second dialog turn information, and outputting one selected from the first dialog turn information and the second dialog turn information when the first dialog turn information is identical to the second dialog turn information;

searching the weight database for weights corresponding to the synthesized dialog turn information, assigning a positive weight to each of the first and second dialog turn information based on the synthesized dialog turn information and selecting whichever one of the first and second dialog turn information has a greater positive weight; and

determining a final dialog turn using the selected one dialog turn information for controlling a dialog between the user and the spoken dialog system.

17. The method of claim 16, wherein the determining the final dialog turn further comprising using information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user to determine the final dialog turn.

18. The method of claim 16, wherein the generating the first dialog turn information comprises:

recognizing the user speech and analyzing the user speech into speech elements;

extracting sentence pattern information, modal information, and discourse marker information using the speech elements; and

generating the first dialog turn information using the sentence pattern information, the modal information, and the discourse marker information.

19. The method of claim 18, wherein the speech elements comprise morphemes.

20. The method of claim 18, wherein the generating the first dialog turn information comprises, when at least one dialog turn information corresponding to the sentence pattern information, the modal information, and the discourse marker information exist, generating the first dialog turn information with reference to a recent dialog history.

21. The method of claim 16, wherein the generating the second dialog turn information comprises:

obtaining information on a body part pattern from an image of the user;

comparing the obtained information with standard pattern information to generate information on movements of the user; and

generating the second dialog turn information using the movement information.

22. The method of claim 21, wherein the obtaining information on the body part comprises obtaining information on a face, eye, eyebrow, and eyelid pattern from the user image.

23. The method of claim 16, wherein the determining the final dialog turn comprises:

determining from status information on a status of the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determining the final dialog turn depending on information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determining the final dialog turn depending on information on a user no-answer time and the selected one dialog turn information.

24. The method of claim 23, wherein the determining the final dialog turn depending on the information on whether the user speech is input and the selected one dialog turn information comprises:

determining whether the user speech is input;

if it is determined that the user speech has been input, controlling the spoken dialog system so as to stop speaking; and

if it is determined that the user speech has not been input, controlling the spoken dialog system so as to one of stop, keep, and commence speaking selected based on the selected one dialog turn information.

25. The method of claim 24, wherein the controlling the spoken dialog system so as to one of stop, keep, and commence speaking comprises:

if it is determined that the user speech has not been input, a residual amount of an answer being presently spoken by the spoken dialog system is checked, and

if the residual amount of the answer presently spoken is to be finished within a predetermined period of time and the final dialog turn is determined as the turn-take, the spoken dialog is controlled to continue speaking, otherwise the spoken dialog is controlled to stop speaking.

26. The method of claim 23, wherein the determining the final dialog turn depending on the information on the user no-answer time or the selected one dialog turn information comprises:

determining whether the user no-answer time exceeds a predetermined period of time;

if it is determined that the user no-answer time exceeds the predetermined period of time, controlling the spoken dialog system so as to commence speaking; and

if it is determined that the user no-answer time does not exceed the predetermined period of time, controlling the spoken dialog system to one of stop, keep, and commence speaking selected based on the selected one dialog turn information.

27. A computer-readable storage medium encoded with computer readable code to control a computer to implement the method of claim 16.

28. An apparatus for managing a dialog turn between a user and a spoken dialog system including a computer having computing device-executable instructions, the apparatus comprising:

a weight database, controlled by the computer, that stores weights assigned to varying dialog turn information based on statistical information;

a dialog analyzer, controlled by the computer, that generates first dialog turn information using dialog information analyzed from a speech uttered by the user;

a body expression, controlled by the computer, analyzer that generates second dialog turn information using body expression information analyzed from the user;

a synthesizer, controlled by the computer, that synthesizes the first dialog turn information and the second dialog turn information, the synthesizer outputting the synthesized dialog turn information to a weight searcher when the first dialog turn information is different than the second dialog turn information and outputting one selected from the first dialog turn information and the second dialog turn information to a dialog controller when the first dialog turn information is identical to the second dialog turn information;

the weight searcher searching the weight database for weights corresponding to the synthesized dialog turn information output from the synthesizer, assigning a positive weight to each of the first and second dialog turn information based on the synthesized dialog turn information and selecting whichever one of the first and second dialog turn information has a greater positive weight; and

the dialog controller determining a final dialog turn using the selected one dialog turn information for controlling a dialog between the user and the spoken dialog system.

29. The apparatus of claim 28, wherein the dialog controller further determines the final dialog turn using information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user.

30. The apparatus of claim 28, wherein the dialog analyzer comprises:

an analyzer that analyzes the user speech into speech elements;

a keyword extractor that extracts sentence pattern information, modal information, and discourse marker information using the speech elements;

a sentence pattern database that stores information on speech acts and dialog turns resulting from the sentence pattern information, the modal information, and the discourse marker information; and

a sentence pattern searcher that searches the sentence pattern database using the sentence pattern information, the modal information, and the discourse marker information to generate the first dialog turn information.

31. The apparatus of claim 30, wherein the speech element analyzed by the analyzer includes morphemes.

32. The apparatus of claim 30, wherein the dialog analyzer further comprises:

a dialog history database that stores information on speech acts of a recent dialog sentence; and

a candidate dialog turn selector that selects the first dialog turn information with reference to the dialog history database when the sentence pattern searcher searches out at least one dialog turn information from the sentence pattern database.

33. The apparatus of claim 28, wherein the body expression analyzer comprises:

an image normalizer that normalizes an image of the user so as to have a predetermined size and color;

a body pattern database that stores data on feature points for recognition of the body pattern;

a body pattern searcher that searches the body pattern database to obtain information on locations of the body pattern of the normalized body image;

a movement information generator that compares the obtained position information with information on locations of standard patterns to generate movement information;

a body expression pattern database that stores information on body expression patterns corresponding to the determined movement information and dialog turns of current speakers; and

a body expression pattern searcher that searches the body expression pattern database using the movement information to generate the second dialog turn information.

34. The apparatus of claim 33, wherein the body pattern comprises face, eye, eyebrow, and eyelid patterns.

35. The apparatus of claim 28, wherein the dialog controller:

determines from status information on the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determines the final dialog turn based on the information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determines the final dialog turn based the user no-answer time and the selected one dialog turn information.

36. A robot using the apparatus of claim 28.