Method and system for automation of response selection and composition in dialog systems

• US 10,311,869 B2
• Filed: 10/21/2015
• Issued: 06/04/2019
• Est. Priority Date: 10/21/2014
• Status: Active Grant

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Weng, Fuliang, Shen, Zhongnan
• Primary Examiner(s): Guerra-Erazo, Edgar X
• Application Number:
  US15/507,374
• Publication Number:
  US 20180240459A1
• Time in Patent Office: 1,322 Days
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 25/63 for estimating an emotional state
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A dialog system includes a processor. The system can further include a dialog manager. The dialog manager can be configured to receive input from a user using the processor. The system can further include a user category classification and detection module, which is configured to identify categories for the user from the received input. The system can further include a user mood detection and tracking module configured to identify a mood of the user. The system can further include a user physical and mind state and energy level detection module configured to identify a mental status of the user. The system can further include a user acquaintance module configured to identify an acquaintance status of the user. The system can further include user personality detection and tracking module configured to identify a personality status of the user. The system can further include a conversational context detection and response generation module.

Claims

1. A system comprising:

a processor;

a dialog manager configured to receive input from a user, the received input being the user'"'"'s speech;

a user category classification and detection module configured to use the processor to identify categories for the user with reference to the received input;

a user mood detection and tracking module configured to use the processor to identify a mood of the user with reference to the received input;

a user physical and mind state and energy level detection module configured to use the processor to identify at least one of a physical status and a mental status of the user with reference to the received input;

a user acquaintance module configured to use the processor to identify an acquaintance status of the user with reference to the received input;

a user personality detection and tracking module configured to use the processor to identify a personality status of the user with reference to the received input;

a conversational context detection and management module configured to use the processor to identify a conversational context of the received input; and

a response generation module configured to use the processor to build a knowledge base and generate a response for the user with reference to the received input and to vocalize the response for the user, the categories for the user, the mood of the user, the mental status of the user, the acquaintance status of the user, the personality status of the user, and the conversational context of the received input,

wherein the user category classification and detection module is further configured to associate at least one of an age, a gender, a profession and a relationship with the user with reference to at least one of a voice characteristic of the user, at least one image of the user, and at least one video of the user, and

wherein the response generation module is configured to assign a voice type for vocalizing the response and to select at least one voice characteristic for the voice type based at least in part on the at least one of the age, the gender, the profession, and the relationship associated with the user.

5. The system of claim 1, wherein the user mood detection and tracking module is further configured to associate at least one current mood with a user, the at least one current mood identified with reference to at least one of a current voice characteristic of the user, a current facial expression of the user in at least one image of the user, and a current facial expression of the user in at least one video of the user.

6. The system of claim 1, wherein the user physical and mind state and energy level detection module is further configured to associate at least one of a physical state and a mental state with the user with reference to at least one of a current tone of voice of the user, a current voice pace of the user, a time of day, a user sleep pattern, and the received input.

7. The system of claim 1, wherein:

the acquaintance status is indicative of a familiarity of the user with the system, and

the user acquaintance module is further configured to identify an acquaintance status for the user with reference to data indicating at least one of whether the user has used the system, a number of times the user has used the system, a frequency of user interaction with the system, and an identification of features of the system previously used by the user.

8. The system of claim 1, wherein the user personality detection and tracking module is further configured to monitor interactions between the user and the system over time and to identify the personality status with reference to a voice characteristic of the user on a day-to-day basis and over time, the voice characteristic including at least one of tone of voice, voice pace, words used, features of the system used, number of interactions with the system, and length of interactions with the system.

9. The system of claim 1, wherein the conversational context detection and management module is further configured to identify the conversational context with reference to at least one of an interaction setting, tone of voice of the user, urgency level of a voice of the user, and prior interactions with the user.

2. The system of concept 1, further comprising:

a database, the knowledge base stored in the database.

3. The system of concept 1, wherein the knowledge base includes at least one of a rich word dictionary and a rich expression dictionary.

4. The system of concept 1, wherein the response generation module is further configured to generate the response by:

selecting words with reference to the received input, the categories for the user, the mood of the user, the mental status of the user, the acquaintance status of the user, the personality status of the user, and the conversational context of the received input;

selecting a tone based on the received input, the categories for the user, the mood of the user, the mental status of the user, the acquaintance status of the user, the personality status of the user, and the conversational context of the received input; and

selecting a volume level based on the received input, the categories for the user, the mood of the user, the mental status of the user, the acquaintance status of the user, the personality status of the user, and the conversational context of the received input.

10. A method of operating a dialog system configured to interact with a user using spoken dialog, the method comprising:

receiving voice input into the system via a voice detection system, the voice input indicative of a spoken statement spoken of a user;

using a user category classification and detection module to;

determine at least one voice characteristic of the voice input; and

identify at least one of an age, a gender, and an identity of the user with reference to the at least one voice characteristic;

using a user mood detection and tracking module to associate at least one mood with the user with reference to the at least one voice characteristic;

using a physical and mind state and energy level detection module to associate at least one of a physical state and a mental state with the user with reference to the at least one voice characteristic;

using a user acquaintance module to associate an acquaintance status with the user with reference to the at least one voice characteristic and to a history of prior interaction of the user with the system;

using a user personality detection and tracking module to associate a personality status with the user with reference to the at least one voice characteristic and to the history;

using a conversational context detection and management module to associate a conversational context with the voice input with reference to the at least one voice characteristic;

generating a response to the spoken statement and assigning a voice type to the response, the voice type having at least one voice characteristic that is selected with reference to the category, the mood, the at least one of physical and mental state, the personality, and the conversational context associated with the user; and

using the system to vocalize the response with the assigned voice type having the corresponding voice characteristics,

wherein the user category classification and detection module is further configured to associate at least one of an age, a gender, a profession and a relationship with the user with reference to at least one of a voice characteristic of the user, at least one image of the user, and at least one video of the user, and

wherein the response generation module is configured to assign a voice type for vocalizing the response and to select at least one voice characteristic for the voice type based at least in part on at least one of the age, the gender, the profession, and the relationship associated with the user.