Conversation control apparatus, conversation control method, and programs therefor

• US 7,676,369 B2
• Filed: 11/19/2004
• Issued: 03/09/2010
• Est. Priority Date: 11/20/2003
• Status: Active Grant

• Current Assignee: Universal Entertainment Corporation
• Original Assignee: Universal Entertainment Corporation
• Inventors: Katukura, Hiroshi, Huang, Shengyang, Fujimoto, Jun
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): GODBOLD, DOUGLAS
• Application Number:
  US10/993,884
• Publication Number:
  US 20050144013A1
• Time in Patent Office: 1,936 Days
• Field of Search: 704/9, 704/231, 704/257, 704/258, 704/270, 704/275
• US Class Current: 704/270
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology

Abstract

The present invention is a conversation control apparatus for carrying out conversation control based on both speech content and conversation state and information acquired from outside, and carrying out natural responses. This apparatus comprises a conversation database storing a plurality of items of conversation topic specifying information, a conversation control unit for selecting a reply sentence using conversation topic specifying information determined using the conversation history and conversation information, and an environment recognition unit for acquiring environment recognition information, wherein the environment recognition unit generates event information based on acquired environment recognition information, and the conversation control unit selects a reply sentence based on the event information.

Claims

1. A conversation control apparatus, comprising:

(a) a conversation database having stored therein;

a plurality of topic specifying information items;

a plurality of topic titles including sub-pluralities respectively correlated to correspond to respective ones of said topic specifying information items;

a plurality of reply sentences including sub-pluralities each respectively correlated to correspond to a respective one of said topic titles; and

a plurality of event information flags each corresponding to an emotion and including sub-pluralities each correlated to correspond to a respective one of said reply sentences;

(b) a voice input unit configured to receive speech input of a user;

(c) a sensor unit configured to acquire facial image data of the user;

(d) an emotion estimation module configured to estimate a current emotion of the user, based upon a characteristic quantity of an expression computed from the facial image data of the user acquired by the sensor unit, and to generate event information indicative of a result of the estimate;

(e) a past conversation information storage unit storing a plurality of past conversation information items determined based upon a past speech by the user and a past reply sentence in response to the past speech, the past reply sentence having been output by the conversation control apparatus;

(f) an output unit configured to output sentences; and

(g) a conversation control unit, the conversation control unit being configured to execute the following operations;

(i) accept the speech input received by the voice input unit from the user as current conversation information and store the current conversation information for future use as the past conversation information of the user in the past conversation information storage unit;

(ii) acquire the facial image data of the user, who uttered the speech input, and generate by the emotion estimation module, the event information used for estimating the current emotion of the user, based upon the acquired facial image data of the user;

(iii) extract a relevant conversation information item, from among the plurality of the past conversation information items stored in the past conversation information storage unit, based upon the current conversation information of the user accepted in operation (i);

(iv) extract a relevant topic specifying information item, from among the plurality of the topic specifying information items stored in the conversation database unit, based upon the relevant conversation information item extracted in the operation (iii);

(v) extract a relevant topic title, from among the plurality of the topic titles determined as relevant based on corresponding to the relevant topic specifying information item extracted in the operation (iv) which was extracted based on the current conversation information of the user input in the operation (i), and also to select one of the sub-plurality of reply sentences by determining correlation thereof to the relevant topic title;

(vi) extract a relevant event information flag, from among the sub-plurality of the event information flags correlated to the selected one of the sub-plurality of reply sentences correlated to the relevant topic tide extracted in the operation (v), based upon the event information indicative of the current emotion of the user and generated in the operation (ii) by the emotion estimation module;

(vii) extract a relevant reply sentence from the sub-plurality of reply sentences correlated to the relevant topic title extracted in the operation (v), by determining the relevant reply sentence corresponds to the relevant event information flag extracted in the operation (vi), such that said relevant reply sentence is extracted based upon all of the following;

the current conversation information of the user accepted in operation (i) being used to extract the relevant conversation information item which in turn is used to extract the relevant topic specifying information item which is then used to extract the relevant topic title which is then used to select the sub-plurality of reply sentences;

the past speech by the user and the past reply sentence issued in response to the past speech being used to provide the past conversation information from which the relevant conversation information item is extracted; and

outside information in the form of the facial image data of the user based upon which the event information is generated and used to extract the relevant reply sentence from the selected sub-plurality of reply sentences by confirming the event information flag of the reply sentence relates to the event information; and

(viii) output the relevant reply sentence, extracted in the operation (vii), to the user.

2. The conversation control apparatus according to claim 1, further comprising:

an emotional condition information management unit configured to store emotional condition information of a predetermined character;

a display unit for displaying the predetermined character;

the emotional condition information management unit being further configured to;

receive the event information indicative of the current emotion of the user generated in operation (ii) by the emotion estimation module; and

update the emotional condition information of the predetermined character so that the current emotion of the user is reflected in the predetermined character, based upon the event information received and indicative of the current emotion of the user; and

the display unit being configured to display a motion and an expression of the predetermined character as a function of the emotional condition information last updated by the emotional condition information management unit.

4. The method according to claim 1, further comprising:

storing emotional condition information of a predetermined character;

receive the event information indicative of the current emotion of the user generated in operation (ii); and

updating the emotional condition information of the predetermined character so that the current emotion of the user is reflected in the predetermined character, based upon the event information received and indicative of the current emotion of the user; and

displaying the predetermined character on a display unit, said displaying including displaying a motion and an expression of the predetermined character as a function of the emotional condition information last updated.

3. A method of effecting conversation control using a conversation control apparatus, comprising:

(a) providing a conversation database having stored therein;

a plurality of topic specifying information items;

a plurality of topic titles including sub-pluralities respectively correlated to correspond to respective ones of said topic specifying information items;

a plurality of reply sentences including sub-pluralities each respectively correlated to correspond to a respective one of said topic tides; and

a plurality of event information flags each corresponding to an emotion and including sub-pluralities each correlated to correspond to a respective one of said reply sentences;

(b) providing a voice input unit configured to receive speech input of a user;

(c) a sensor unit configured to acquire facial image data of the user;

(d) providing an emotion estimation module configured to estimate a current emotion of the user, based upon a characteristic quantity of an expression computed from the facial image data of the user acquired by the sensor unit, and to generate event information indicative of a result of the estimate;

(e) providing a past conversation information storage unit storing a plurality of past conversation information items determined based upon a past speech by the user and a past reply sentence in response to the past speech, the past reply sentence having been output by the conversation control apparatus;

(f) providing an output unit configured to output sentences; and

(g) executing the following operations;

(i) accepting speech input received by the voice input unit from the user as current conversation information and storing the current conversation information for future use as the past conversation information of the user in the past conversation information storage unit;

(ii) acquiring the facial image data of the user, who uttered the speech input, and generating by the emotion estimation module, the event information used for estimating the current emotion of the user, based upon the acquired facial image data of the user;

(iii) extracting a relevant conversation information item from among the plurality of the past conversation information items stored in the past conversation information storage unit, based upon the current conversation information of the user accepted in operation (i);

(iv) extracting a relevant topic specifying information item from among the plurality of the topic specifying information items stored in the conversation database unit, based upon the relevant conversation information item extracted in the operation (iii);

(v) extracting a relevant topic tide from among the plurality of the topic titles by determining relevancy based on correspondence to the relevant topic specifying information item extracted in the operation (iv) which was extracted based on the current conversation information of the user input in the operation (i), and also selecting one of the sub-plurality of reply sentences by determining correlation thereof to the relevant topic title;

(vi) extracting a relevant event information flag, from among the sub-plurality of the event information flags correlated to the selected one of the sub-plurality of reply sentences correlated to the relevant topic title extracted in the operation (v), based upon the event information indicative of the current emotion of the user and generated in the operation (ii) by the emotion estimation module;

(vii) extracting a relevant reply sentence from the sub-plurality of reply sentences correlated to the relevant topic title extracted in the operation (v), by determining the relevant reply sentence corresponds to the relevant event information flag extracted in the operation (vi), such that said relevant reply sentence is extracted based upon all of the following;

the current conversation information of the user accepted in operation (i) being used to extract the relevant conversation information item which in turn is used to extract the relevant topic specifying information item which is then used to extract the relevant topic title which is then used to select the sub-plurality of reply sentences;

the past speech by the user and the past reply sentence issued m response to the past speech being used to provide the past conversation information from which the relevant conversation information item is extracted; and

outside information in the form of the facial image data of the user based upon which the event information is generated and used to extract the relevant reply sentence from the selected sub-plurality of reply sentences by confirming the event information flag of the reply sentence relates to the event information; and

(viii) outputting the relevant reply sentence, extracted in the operation (vii), to the user.