Dialogue system and method for responding to multimodal input using calculated situation adaptability

• US 8,719,015 B2
• Filed: 12/10/2010
• Issued: 05/06/2014
• Est. Priority Date: 12/11/2009
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Jang, Jun Won, Han, Woo Sup
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US12/964,866
• Publication Number:
  US 20110144999A1
• Time in Patent Office: 1,243 Days
• Field of Search: 704231-257
• US Class Current: 704/231
• CPC Class Codes: G06F 16/60 of audio data
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/00 Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 muting semiconductor-based amplifiers when some special characteristics of a signal are sensed by a speech detector, e.g. sensing when no signal is present, H03G3/34

Abstract

A dialogue system and a method for the same are disclosed. The dialogue system includes a multimodal input unit receiving speech and non-speech information of a user, a domain reasoner, which stores a plurality of pre-stored situations, each of which is formed by a combination one or more speech and non-speech information, calculating each adaptability of the pre-stored situations on the basis of a generated situation based on the speech and the non-speech information received from the multimodal input unit, and determining a current domain according to the calculated adaptability, a dialogue manager to select a response corresponding to the current domain, and a multimodal output unit to output the response. The dialogue system performs domain reasoning using a situation including information combinations reflected in the domain reasoning process, current information, and a speech recognition result, and reduces the size of a dialogue search space while increasing domain reasoning accuracy.

Claims

1. A dialogue system, comprising:

a multimodal input unit receiving speech and non-speech information of a user;

a domain reasoner, which stores a plurality of pre-stored situations, each of which is formed by a combination one or more speech and non-speech information, calculating an adaptability of each of the pre-stored situations based on a comparison with a situation generated based on the speech and the non-speech information received from the multimodal input unit, and determining a current domain according to the calculated adaptability of each of the pre-stored situations;

a dialogue manager selecting a response corresponding to the current domain; and

a multimodal output unit outputting the response,

wherein the non-speech information comprises at least one of facial expression, dress, and gesture belonging to user status information as an input image.

2. The dialogue system according to claim 1, wherein the domain reasoner generates the situation by combining the speech and the non-speech information received from the multimodal input unit, compares the generated situation with the pre-stored situations, and calculates the adaptability of each of the pre-stored situations according to the result of comparison.

3. The dialogue system according to claim 2, wherein the domain reasoner receives a user response corresponding to the output response to determine a satisfaction degree of the user, reflects the satisfaction degree in the generated situation, updates the reflected result, and stores the updated result.

4. The dialogue system according to claim 1, wherein the pre-stored situations are generated by a learning operation and are stored.

5. The dialogue system according to claim 1, wherein each of the pre-stored situations is configured by combining preference-, habit-, and schedule-information of the user, at least one form of non-speech information from among facial expression, dress, and gesture belonging to user status information, and speech information of the user, and each of the pre-stored situations has a domain corresponding to the combination result.

6. The dialogue system according to claim 1, wherein the multimodal input unit receives at least one form of non-speech information from among facial expression, dress, and gesture belonging to user status information as an input image.

7. The dialogue system according to claim 1, wherein the domain reasoner selects one of the pre-stored situations having the highest adaptability.

8. The dialogue system according to claim 7, further comprising:

a domain database (DB) unit storing a plurality of domains,

wherein the domain reasoner calculates an adaptability of each domain using the selected situation, and determines a domain having a highest adaptability to be a current domain.

9. The dialogue system according to claim 1, wherein:

the multimodal input unit detects environmental information of a room in which the user is currently located, and

the domain reasoner determines the current domain using the environmental information.

10. The dialogue system according to claim 1, further comprising:

an information provider providing schedule information of the user, behavior history information, indoor structure information, and weather information,

wherein the domain reasoner determines the current domain using at least one of a schedule of the user, a behavior history, an indoor structure, and the weather information.

11. The dialogue system according to claim 1, wherein:

the dialogue manager transmits dialogue context and previous dialogue information to the domain reasoner; and

the domain reasoner determines the current domain using at least one of the dialogue context and previous dialogue information.

12. A dialogue system, comprising:

a multimodal input unit receiving speech and non-speech information of a user;

a domain reasoner, which stores a plurality of pre-stored situations, each of which is formed by a combination one or more speech and non-speech information, selecting a candidate domain corresponding to a speech signal received from the multimodal input unit, calculating an adaptability of each of the pre-stored situations based on a comparison of a situation generated based on the non-speech information received from the multimodal input unit, and determining a current domain based on one of the pre-stored situations having the highest adaptability in relation to the generated situation, and the candidate domain;

a dialogue manager selecting a response corresponding to the current domain; and

a multimodal output unit outputting the selected response.

wherein the non-speech information comprises at least one of facial expression, dress, and gesture belonging to user status information as an input image.

13. The dialogue system according to claim 12, wherein each of the pre-stored situations includes a related domain.

14. A dialogue method for use in a dialogue system, the method comprising:

receiving speech information and non-speech information from a user;

calculating on a processor, an adaptability of each of pre-stored situations based on a comparison to a generated situation based on the received speech and non-speech information;

determining a current domain based on the calculated adaptability; and

selecting a response corresponding to the determined current domain and outputting the selected response,

wherein the non-speech information comprises at least one of facial expression, dress, and gesture belonging to user status information as an input image.

15. The dialogue method according to claim 14, wherein each of the pre-stored situations is configured by combining preference-, habit-, and schedule-information of the user, and at least one form of non-speech information from among facial expression, dress, and gesture belonging to user status information, and speech information of the user.

16. The dialogue method according to claim 14, wherein the determining of the current domain includes:

generating the situation by combining the received speech information and the received non-speech information;

calculating the adaptability by comparing the generated situation with each of the pre-stored situations; and

selecting one of the pre-stored situations having the highest calculated adaptability.

17. The dialogue method according to claim 16, wherein the determining of the current domain further includes:

receiving a user response corresponding to the response output when the current domain is determined; and

reflecting a satisfaction degree corresponding to the user response in the generated situation, and storing the reflected result.

18. The dialogue method according to claim 16, wherein the determining of the current domain further includes:

calculating an adaptability of each of pre-stored domains by applying the selected situation to each of the pre-stored domains; and

determining a specific domain having a highest adaptability from among the pre-stored domains to be a current domain.

19. The dialogue method according to claim 18, wherein the determining of the specific domain having the highest adaptability to be a current domain includes:

comparing the adaptability of the specific domain having the highest adaptability with a reference value;

determining the specific domain to be the current domain when the adaptability is higher than the reference value; and

determining an occurrence of a failure in current domain reasoning, and re-requesting a dialogue, when the adaptability is equal to or lower than the reference value.

20. The dialogue method according to claim 14, wherein the determining of the current domain includes:

receiving environment information of a room in which the user is currently located; and

determining the current domain using the environmental information.

21. The dialogue method according to claim 14, wherein the determining of the current domain includes:

receiving schedule information of the user, behavior history information, indoor structure information, and weather information; and

determining the current domain using at least one of schedule information of the user, behavior history information, indoor structure information, and the weather information.

22. The dialogue method according to claim 14, wherein the determining of the current domain includes:

receiving dialogue context and previous dialogue information as an input; and

determining the current domain using at least one of the dialogue context and the previous dialogue information.

23. The dialogue method according to claim 14, wherein each of the pre-stored situations is configured by a combination of at least one form of non-speech information, and wherein a candidate domain is selected by recognizing the speech information, calculating an adaptability of each of the pre-stored situations based on the received non-speech information, selecting a specific situation having the highest adaptability from among the pre-stored situations, and determining a current domain based on the domain corresponding to the selected situation and the candidate domain.