Dialogue control device, dialogue control method and non-transitory computer-readable information recording medium

• US 9,953,078 B2
• Filed: 06/30/2016
• Issued: 04/24/2018
• Est. Priority Date: 09/01/2015
• Status: Active Grant

• Current Assignee: Casio Computer Company Limited
• Original Assignee: Casio Computer Company Limited
• Inventors: Kawamura, Yoshihiro
• Primary Examiner(s): SHIN, SEONG-AH A
• Application Number:
  US15/198,904
• Publication Number:
  US 20170060839A1
• Time in Patent Office: 663 Days
• Field of Search: 704 9, 704231, 704273, 715727, 715765
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/227 of the speaker; Human-factor methodology
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

An emotion estimator of a dialogue control device estimates a dialogue partner'"'"'s emotion based on the dialogue partner'"'"'s facial expression or voice prosody. A dialogue controller controls the dialogue with the dialogue partner using a topic selected based on whether the emotion estimated by the emotion estimator is good or bad. Specifically, the dialogue controller controls the dialogue with the dialogue partner based on a topic map created by a topic mapper and including a topic that the dialogue partner likes.

Claims

1. A dialogue control device for undertaking a dialogue with a dialogue partner comprising a controller, an inputter including a microphone and a camera, an outputter, and a storage,

wherein the storage includes a topic map, the topic map including a plurality of topics each associated with a corresponding score that is kept and updated on a real-time basis during the dialogue with the dialogue partner to provide a numerical measure of a level of interest of a the dialogue partner in the topics, the topic map being an n-dimensional topic map with the topics being mapped according to n-dimensional feature vectors that represent multiple characteristics of each of the topics, and the storage further including a text database for storing words associated with emotions, a facial expression database for storing template images of facial expressions associated with emotions, and a prosody database for storing prosodies associated with emotions, and

wherein the controller

selects one of the topics from the topic map and causes the outputter to output the selected topic,

estimates emotion of the dialogue partner based on an utterance of the dialogue partner with respect to the outputted topic picked up by the microphone and converted to text which is referenced to the stored words in the text database, a facial expression of the dialogue partner during the utterance which is determined from a facial image of the dialogue partner taken by the camera and matched to the stored template images in the facial expression database, and the prosody of the utterance referenced to the stored prosodies in the prosody database,

adds to the score associated with the selected topic if the estimated emotion is good, and

subtracts from the score associated with the selected topic if the estimated emotion is bad, and switches the selected topic to another topic that is selected based on the feature vector thereof to improve the bad emotion of the dialogue partner.

2. The dialogue control device according to claim 1,

wherein the topic map is arranged on an external server, and

wherein the controller accesses the external server each time the controller references the topic map.

3. The dialogue control device according to claim 1, wherein the controller initially develops the topic map by searching likes and tastes of the dialogue partner.

4. The dialogue control device according to claim 1, wherein the controller predicts a likeable topic for the dialogue partner from the topic map, and causes the outputter to output the predicted topic.

5. A method for controlling a dialogue performed by a dialogue control device including a controller, an inputter including a microphone and a camera, and an outputter, the method comprising:

creating a topic map including a topic associated with a score that shows a level of interest of a dialogue partner in the topic, the topic map being mapped according to characteristics of the topic;

storing a text database for storing words associated with emotions, a facial expression database for storing template images of facial expressions associated with emotions, and a prosody database for storing prosodies associated with emotions; and

using a controller to perform the following steps;

selecting the topic from the topic map, and causing the outputter to output the selected topic;

estimating emotion of the dialogue partner based on an utterance of the dialogue partner with respect to the outputted topic picked up by the microphone and converted to text which is referenced to the stored words in the text database, a facial expression of the dialogue partner during the utterance which is determined from a facial image of the dialogue partner taken by the camera and matched to the stored template images in the facial expression database, and the prosody of the utterance referenced to the stored prosodies in the prosody database;

adding to the score associated with the selected topic if the estimated emotion is good; and

subtracting from the score associated with the selected topic if the estimated emotion is bad, and switching the selected topic to another topic that is selected based on the score thereof for improving the bad emotion of the dialogue partner.

6. A non-transitory computer-readable information recording medium that records a program for a computer of a dialogue control device that includes a controller, an inputter including a microphone and a camera, and an outputter, and has stored a text database for storing words associated with emotions, a facial expression database for storing template images of facial expressions associated with emotions, and a prosody database for storing prosodies associated with emotions, the program causing the controller to:

create a topic map that includes a topic associated with a score showing a level of interest of a dialogue partner in the topic, the topic map being mapped according to characteristics of the topic;

select the topic from the topic map, and cause the outputter to output the selected topic;

estimate emotion of the dialogue partner based on an utterance of the dialogue partner with respect to the outputted topic picked up by the microphone and converted to text which is referenced to the stored words in the text database, a facial expression of the dialogue partner during the utterance which is determined from a facial image of the dialogue partner taken by the camera and matched to the stored template images in the facial expression database, and a prosody of the utterance referenced to the stored prosodies in the prosody database;

add to the score associated with the selected topic if the estimated emotion is good; and

subtract from the score associated with the selected topic if the estimated emotion is bad, and switch the selected topic to another topic that is selected based on the score thereof for improving the bad emotion of the dialogue partner.