Method of and apparatus for managing dialog between user and agent

• US 20040122673A1
• Filed: 12/11/2003
• Published: 06/24/2004
• Est. Priority Date: 12/11/2002
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Hong, Young-Jin, Lee, Jae-Won, Park, Joon-Ah
• Granted Patent: US 7,734,468 B2
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

A method of rapidly and precisely managing a dialog turn between a user and an agent by using speech information, facial expression information, and delay time information includes generating first dialog turn information using dialog information analyzed from a speech uttered by the user, generating second dialog turn information using facial expression information analyzed from a face image of the user, and determining a final dialog turn using the first and second dialog turn information, information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user.

Claims

1. A method of managing a dialog turn between a user and a spoken dialog system, the method comprising:

generating first dialog turn information using dialog information analyzed from a speech uttered by the user;

generating second dialog turn information using facial expression information analyzed from a face image of the user;

selecting one of the first and second dialog turn information using predetermined weight information; and

determining a final dialog turn using the selected one dialog turn information, information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user, for controlling a dialog between the user and the spoken dialog system.

2. The method of claim 1, wherein the generating the first dialog turn information comprises:

recognizing the user speech and analyzing the user speech into morphemes;

extracting sentence pattern information, modal information, and discourse marker information using the morphemes; and

generating the first dialog turn information using the sentence pattern information, the modal information, and the discourse marker information.

3. The method of claim 2, wherein the generating the first dialog turn information further comprises, when at least one dialog turn information corresponding to the sentence pattern information, the modal information, and the discourse marker information exist, generating the first dialog turn information with reference to a recent dialog history.

4. The method of claim 1, wherein the generating the second dialog turn information comprises:

obtaining information on face, eye, eyebrow, and eyelid patterns from the user face image;

comparing the obtained information with standard pattern information to generate information on movements of the user; and

generating the second dialog turn information using the movement information.

5. The method of claim 1, wherein the determining a final dialog turn comprises:

determining from the status information on the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determining the final dialog turn depending on the information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determining the final dialog turn depending on the information on the user no-answer time or the selected one dialog turn information.

6. The method of claim 5, wherein the determining the final dialog turn depending on the information on whether the user speech is input and the selected one dialog turn information comprises:

determining whether the user speech is input;

if it is determined that the user speech has been input, controlling the spoken dialog system so as to stop speaking; and

if it is determined that the user speech has not been input, controlling the spoken dialog system so as to one of stop, keep, and commence speaking determined based on the selected one dialog turn information.

7. The method of claim 6, wherein the controlling the spoken dialog system so as to stop, keep, or commence speaking based on the selected one dialog turn information comprises:

if it is determined that the user speech has not been input, a residual amount of an answer being presently spoken by the spoken dialog system is checked, and

if the residual amount of the answer being presently spoken is to be finished within a predetermined period of time, the final dialog turn is determined depending on the selected one dialog turn information.

8. The method of claim 5, wherein the determining the final dialog turn depending on the information on the user no-answer time or the selected one dialog turn information comprises:

determining whether the user no-answer time exceeds a predetermined period of time;

if it is determined that the user no-answer time exceeds the predetermined period of time, controlling the spoken dialog system so as to commence speaking; and

if it is determined that the user no-answer time does not exceed the predetermined period of time, controlling the spoken dialog system to one of stop, keep, and commence speaking determined based on the selected one dialog turn information.

9. A computer-readable recording medium encoded with processing instructions for implementing the method of claim 1 performed by a computer.

10. An apparatus for managing a dialog turn between a user and a spoken dialog system, the apparatus comprising:

a dialog analyzer that generates first dialog turn information using dialog information analyzed from a speech uttered by the user;

a facial expression analyzer that generates second dialog turn information using facial expression information analyzed from a face image of the user;

a dialog turn determiner that selects one of the first and second dialog turn information using predetermined weight information; and

a dialog controller that determines a final dialog turn using the selected one dialog turn information, information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user, for controlling a dialog between the user and the spoken dialog system.

11. The apparatus of claim 10, wherein the dialog analyzer comprises:

a morpheme analyzer that analyzes the user speech into morphemes;

a keyword extractor that extracts sentence pattern information, modal information, and discourse marker information using the morphemes;

a sentence pattern database that stores information on speech acts and dialog turns resulting from the sentence pattern information, the modal information, and the discourse marker information; and

a sentence pattern searcher that searches the sentence pattern database using the sentence pattern information, the modal information, and the discourse marker information to generate the first dialog turn information.

12. The apparatus of claim 11, wherein the dialog analyzer further comprises:

a dialog history database that stores information on speech acts of a recent dialog sentence; and

a candidate dialog turn selector that selects the first dialog turn information with reference to the dialog history database when the sentence pattern searcher searches out at least one dialog turn information from the sentence pattern database.

13. The apparatus of claim 10, wherein the facial expression analyzer comprises:

an image normalizer that normalizes the user face image so as to have a predetermined size and color;

a face pattern database that stores data on feature points for recognition of face, eye, eyebrow, and eyelid patterns;

a face pattern searcher that searches the face pattern database to obtain information on locations of face, eye, eyebrow, and eyelid patterns of the normalized face image;

a movement information generator that compares the obtained position information with information on locations of standard patterns to generate movement information;

a facial expression pattern database that stores information on facial expression patterns corresponding to the determined movement information and dialog turns of current speakers; and

a facial expression pattern searcher that searches the facial expression pattern database using the movement information to generate the second dialog turn information.

14. The apparatus of claim 10, wherein the dialog controller:

determines from the status information on the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determines the final dialog turn based on the information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determines the final dialog turn based the user no-answer time and the selected one dialog turn information.

15. A spoken dialog system comprising:

a speech input unit that amplifies a user speech input via a microphone to a predetermined level;

a speech recognizer that recognizes the speech signal provided by the speech input unit to output a character string signal;

an image input unit that inputs a user face image;

a timer counts a user no-answer time for a predetermined period of time;

a dialog manager that generates first dialog turn information using dialog information analyzed from the character string signal, generates second dialog turn information using facial expression information analyzed from the user face image, and determines a final dialog turn using the first and second dialog turn information, information on a status of the spoken dialog system, the character string signal, and the user no-answer time; and

an answer generator that generates an answer corresponding to the character string signal with reference to a dialog model database according to a control operation of the dialog manager.

16. A method of managing a dialog turn between a user and a spoken dialog system, the method comprising:

generating first dialog turn information using dialog information analyzed from a speech uttered by the user;

generating second dialog turn information using body expression information analyzed from a detected body expression of the user;

selecting one of the first and second dialog turn information using predetermined weight information; and

determining a final dialog turn using the selected one dialog turn information for controlling a dialog between the user and the spoken dialog system.

17. The method of claim 16, wherein the determining the final dialog turn further comprising using information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user to determine the final dialog turn.

18. The method of claim 16, wherein the generating the first dialog turn information comprises:

recognizing the user speech and analyzing the user speech into speech elements;

extracting sentence pattern information, modal information, and discourse marker information using the speech elements; and

generating the first dialog turn information using the sentence pattern information, the modal information, and the discourse marker information.

19. The method of claim 18, wherein the speech elements comprise morphemes.

20. The method of claim 18, wherein the generating the first dialog turn information comprises, when at least one dialog turn information corresponding to the sentence pattern information, the modal information, and the discourse marker information exist, generating the first dialog turn information with reference to a recent dialog history.

21. The method of claim 16, wherein the generating the second dialog turn information comprises:

obtaining information on a body part pattern from an image of the user;

comparing the obtained information with standard pattern information to generate information on movements of the user; and

generating the second dialog turn information using the movement information.

22. The method of claim 21, wherein the obtaining information on the body part comprises obtaining information on a face, eye, eyebrow, and eyelid pattern from the user image.

23. The method of claim 16, wherein the determining the final dialog turn comprises:

determining from status information on a status of the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determining the final dialog turn depending on information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determining the final dialog turn depending on information on a user no-answer time and the selected one dialog turn information.

24. The method of claim 23, wherein the determining the final dialog turn depending on the information on whether the user speech is input and the selected one dialog turn information comprises:

determining whether the user speech is input;

if it is determined that the user speech has been input, controlling the spoken dialog system so as to stop speaking; and

if it is determined that the user speech has not been input, controlling the spoken dialog system so as to one of stop, keep, and commence speaking selected based on the selected one dialog turn information.

25. The method of claim 24, wherein the controlling the spoken dialog system so as to one of stop, keep, and commence speaking comprises:

if it is determined that the user speech has not been input, a residual amount of an answer being presently spoken by the spoken dialog system is checked, and

if the residual amount of the answer presently spoken is to be finished within a predetermined period of time, the final dialog turn is determined depending on the selected one dialog turn information.

26. The method of claim 23, wherein the determining the final dialog turn depending on the information on the user no-answer time or the selected one dialog turn information comprises:

determining whether the user no-answer time exceeds a predetermined period of time;

if it is determined that the user no-answer time exceeds the predetermined period of time, controlling the spoken dialog system so as to commence speaking; and

if it is determined that the user no-answer time does not exceed the predetermined period of time, controlling the spoken dialog system to one of stop, keep, and commence speaking selected based on the selected one dialog turn information.

27. A computer-readable recording medium encoded with processing instructions for implementing the method of claim 16 using a computer.

28. An apparatus for managing a dialog turn between a user and a spoken dialog system, the apparatus comprising:

a dialog analyzer that generates first dialog turn information using dialog information analyzed from a speech uttered by the user;

a body expression analyzer that generates second dialog turn information using body expression information analyzed from the user;

a dialog turn determiner that selects one of the first and second dialog turn information using predetermined weight information; and

a dialog controller that determines a final dialog turn using the selected one dialog turn information for controlling a dialog between the user and the spoken dialog system.

29. The apparatus of claim 28, wherein the dialog controller further determines the final dialog turn using information on a status of the spoken dialog system, information on whether the user speech is input, and information on a no-answer time of the user.

30. The apparatus of claim 28, wherein the dialog analyzer comprises:

an analyzer that analyzes the user speech into speech elements;

a keyword extractor that extracts sentence pattern information, modal information, and discourse marker information using the speech elements;

a sentence pattern database that stores information on speech acts and dialog turns resulting from the sentence pattern information, the modal information, and the discourse marker information; and

a sentence pattern searcher that searches the sentence pattern database using the sentence pattern information, the modal information, and the discourse marker information to generate the first dialog turn information.

31. The apparatus of claim 30, wherein the speech element analyzed by the analyzer includes morphemes.

32. The apparatus of claim 30, wherein the dialog analyzer further comprises:

a dialog history database that stores information on speech acts of a recent dialog sentence; and

a candidate dialog turn selector that selects the first dialog turn information with reference to the dialog history database when the sentence pattern searcher searches out at least one dialog turn information from the sentence pattern database.

33. The apparatus of claim 28, wherein the body expression analyzer comprises:

an image normalizer that normalizes an image of the user so as to have a predetermined size and color;

a body pattern database that stores data on feature points for recognition of the body pattern;

a body pattern searcher that searches the body pattern database to obtain information on locations of the body pattern of the normalized body image;

a movement information generator that compares the obtained position information with information on locations of standard patterns to generate movement information;

a body expression pattern database that stores information on body expression patterns corresponding to the determined movement information and dialog turns of current speakers; and

a body expression pattern searcher that searches the body expression pattern database using the movement information to generate the second dialog turn information.

34. The apparatus of claim 33, wherein the body pattern comprises face, eye, eyebrow, and eyelid patterns.

35. The apparatus of claim 28, wherein the dialog controller:

determines from status information on the spoken dialog system whether the spoken dialog system is presently speaking;

if it is determined that the spoken dialog system is presently speaking, determines the final dialog turn based on the information on whether the user speech is input and the selected one dialog turn information; and

if it is determined that the spoken dialog system is not presently speaking, determines the final dialog turn based the user no-answer time and the selected one dialog turn information.

36. A robot using the apparatus of claim 28.

37. A computer using the apparatus of claim 28.