Dialogue system and method for responding to multimodal input using calculated situation adaptability

• US 9,305,569 B2
• Filed: 04/02/2014
• Issued: 04/05/2016
• Est. Priority Date: 12/11/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jang, Jun Won, Han, Woo Sup
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US14/243,477
• Publication Number:
  US 20140214410A1
• Time in Patent Office: 734 Days
• Field of Search: 704/231, 704/251, 704/275
• US Class Current: 1/1
• CPC Class Codes: G06F 16/60 of audio data
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34

Abstract

A dialogue system and a method for the same are disclosed. The dialogue system includes a multimodal input unit receiving speech and non-speech information of a user, a domain reasoner, which stores a plurality of pre-stored situations, each of which is formed by a combination one or more speech and non-speech information, calculating each adaptability of the pre-stored situations on the basis of a generated situation based on the speech and the non-speech information received from the multimodal input unit, and determining a current domain according to the calculated adaptability, a dialogue manager to select a response corresponding to the current domain, and a multimodal output unit to output the response. The dialogue system performs domain reasoning using a situation including information combinations reflected in the domain reasoning process, current information, and a speech recognition result, and reduces the size of a dialogue search space while increasing domain reasoning accuracy.

Claims

1. A dialogue system, comprising:

a multimodal input unit to receive speech information and non-speech information of a user;

a domain reasoner to generate a situation by combining the speech information and the non-speech information, to compare the generated situation with at least one of a plurality of pre-stored situations, and to determine a current domain corresponding to a speech intention of the user based on the results of comparison;

a dialogue manager to generate a response corresponding to the current domain; and

a multimodal output unit to output the response.

2. The dialogue system according to claim 1, wherein the domain reasoner calculates an adaptability between the generated situation and each of the pre-stored situations based on the results of the comparison,

wherein the adaptability is the coincidence degree acquired by comparing combination information of the generated situation with that of each of the pre-stored situations.

3. The dialogue system according to claim 2, further comprising:

a domain database (DB) unit to store a plurality of domains,

wherein the domain reasoner selects one of the pre-stored situations having the highest calculated adaptability, calculates an adaptability of each of the plurality of domains using the adaptability of selected situation, and determines a domain having the highest adaptability to be a current domain.

4. The dialogue system according to claim 1, wherein each of the pre-stored situations is configured by combining at least one form of non-speech information from among preference-, habit-, and schedule-information of the user, facial expression, dress, and gesture belonging to user status information, and speech information of the user, and each of the pre-stored situations has a domain corresponding to the combination result.

5. The dialogue system according to claim 1, wherein the pre-stored situations are generated by a learning operation and are stored.

6. The dialogue system according to claim 1, wherein the multimodal input unit receives at least one form of non-speech information from among facial expression, dress, and gesture belonging to user status information as an input image.

7. The dialogue system according to claim 1, wherein:

the multimodal input unit detects environmental information of a room in which the user is currently located, and

the domain reasoner determines the current domain based on the environmental information.

8. The dialogue system according to claim 1, further comprising:

an information provider to provide schedule information of the user, behavior history information of the user, indoor structure information, and weather information,

wherein the domain reasoner determines the current domain based on at least one of the schedule information of the user, the behavior history information of the user, the indoor structure information, and the weather information.

9. The dialogue system according to claim 1, wherein:

the dialogue manager transmits dialogue context and previous dialogue information to the domain reasoner; and

the domain reasoner determines the current domain based on at least one of the dialogue context and the previous dialogue information.

10. A dialogue method for use in a dialogue system, the method comprising:

receiving speech information and non-speech information from a user;

generating a situation by combining the speech information and the non-speech information;

comparing the generated situation with at least one of a plurality of pre-stored situations;

determining a current domain corresponding to a speech intention of the user based on the results of comparison; and

generating a response corresponding to the current domain and outputting the response.

11. The dialogue method according to claim 10, wherein the determining of the current domain includes calculating an adaptability between the generated situation and each of the pre-stored situations based on the results of the comparison,

wherein the adaptability is the coincidence degree acquired by comparing combination information of the generated situation with that of each of the pre-stored situations.

12. The dialogue method according to claim 11, wherein the determining of the current domain further includes:

selecting one of the pre-stored situations having the highest calculated adaptability;

calculating an adaptability of each of pre-stored domains using the adaptability of selected situation; and

determining a specific domain having a highest adaptability from among the pre-stored domains to be a current domain.

13. The dialogue method according to claim 12, wherein the determining of the specific domain having the highest adaptability to be a current domain includes:

comparing the adaptability of the specific domain having the highest adaptability with a reference value;

determining the specific domain to be the current domain when the adaptability is higher than the reference value; and

determining an occurrence of a failure in current domain reasoning, and re-requesting a dialogue, when the adaptability is equal to or lower than the reference value.

14. The dialogue method according to claim 10, wherein each of the pre-stored situations is configured by combining at least one form of non-speech information from among preference-, habit-, and schedule-information of the user, facial expression, dress, and gesture belonging to user status information, and speech information of the user.

15. The dialogue method according to claim 10, wherein the received non-speech information is at least one of facial expression, dress, and gesture belonging to user status information.

16. The dialogue method according to claim 10, wherein the determining of the current domain includes:

receiving environment information of a room in which the user is currently located; and

determining the current domain based on the environmental information.

17. The dialogue method according to claim 10, wherein the determining of the current domain includes:

receiving schedule information of the user, behavior history information of the user, indoor structure information, and weather information; and

determining the current domain based on at least one of the schedule information of the user, the behavior history information of the user, the indoor structure information, and the weather information.

18. The dialogue method according to claim 10, wherein the determining of the current domain includes:

receiving dialogue context and previous dialogue information as an input; and

determining the current domain based on at least one of the dialogue context and the previous dialogue information.