Emotion recognizing method, sensibility creating method, device, and software

• US 7,340,393 B2
• Filed: 09/04/2001
• Issued: 03/04/2008
• Est. Priority Date: 09/13/2000
• Status: Active Grant

• Current Assignee: AGI (Meadwestvaco Corporation)
• Original Assignee: Advanced Generation Interface, Inc.
• Inventors: Mitsuyoshi, Shunji
• Primary Examiner(s): ABEBE, DANIEL DEMELASH
• Application Number:
  US10/380,279
• Publication Number:
  US 20030182123A1
• Time in Patent Office: 2,373 Days
• Field of Search: 704/200, 704/207, 704/272
• US Class Current: 704/207
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06N 3/004 Artificial life, i.e. computing arrangements simulating life
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

An object of the invention is to provide an emotion detecting method capable of detecting emotion of a human accurately, and provide sensibility generating method capable of outputting sensibility akin to that of a human. An intensity, a tempo, and intonation in each word of a voice are detected based on an inputted voice signal, amounts of change are obtained for the detected contents, respectively, and signals expressing each states of emotion of anger, sadness, and pleasure are generated based on the amounts of change. A partner'"'"'s emotion or situation information is inputted, and thus instinctive motivation information is generated. Moreover, emotion information including basic emotion parameters of pleasure, anger, and sadness is generated, which is controlled based on the individuality information.

Claims

1. An emotion detecting method for detecting an emotion of a subject, comprising the steps of:

inputting a voice signal;

detecting an intensity of a voice and a tempo expressing speed the voice emerges at, respectively, and detecting, as a time value, intonation expressing an intensity-change pattern in each word the voice makes, based on the inputted voice signal;

obtaining a first amount of change indicating a change in the intensity of the detected voice, in a direction of time axis, a second amount of change indicating a change in the tempo of the voice, in the direction of time axis, and a third amount of change indicating a change in the intonation in the voice, in the direction of time axis, respectively; and

generating signals expressing states of emotion of at least anger, sadness, and pleasure, respectively, based on said first, second, and third amounts of change.

2. An emotion detecting system for detecting an emotion of a subject, comprising:

a voice inputting unit for inputting a voice signal;

an intensity detecting unit for detecting an intensity of a voice based on the voice signal inputted by said voice inputting unit;

a tempo detecting unit for detecting speed the voice emerges at as a tempo based on the voice signal inputted by said voice inputting unit;

an intonation detecting unit for detecting intonation expressing an intensity-change pattern in a word of the voice based on the voice signal inputted by said voice inputting unit;

a change-amount detecting unit for obtaining a first amount of change indicating a change in the intensity of the voice detected by said intensity detecting unit, in a direction of time axis, a second amount of change indicating a change in the tempo of the voice detected by said tempo detecting unit, in the direction of time axis, and a third amount of change indicating a change in the intonation in the voice detected by said intonation detecting unit, in the direction of time axis, respectively; and

an emotion detecting unit for outputting signals expressing states of emotion of at least anger, sadness, and pleasure, respectively, based on the first, second, and third amounts of change detected by said change-amount detecting unit.

3. The emotion detecting system according to claim 2, wherein said intonation detecting unit includes:

a bandpass filter unit for extracting specific frequency components from the voice signal inputted separately for each word;

an area separating unit for separating a power spectrum of the signal extracted by said bandpass filter unit into a plurality of areas based on the intensity of the power spectrum; and

an intonation calculating unit for calculating a value of the intonation based on time intervals between respective centers of the plurality of areas separated by said area separating unit.

4. The emotion detecting system according to claim 2, further comprising:

an imaging unit for receiving image information concerning at least a face of the subject;

an image recognition unit for detecting positional information concerning each part of the face from the image information received by said imaging unit;

an image reference information retaining unit for retaining reference information concerning an amount of characteristic in each part of the face; and

an image characteristic amount detecting unit for detecting an image characteristic amount based on the positional information detected by said image recognition unit and the reference information retained by said image reference information retaining unit, and wherein said emotion detecting unit estimates a state of emotion according to a change in the image characteristic amount detected by said image characteristic amount detecting unit.

5. The emotion detecting system according to claim 2, further comprising:

an emotion information storing unit for sequentially receiving pieces of information concerning the states of emotion detected by said emotion detecting unit and for storing the pieces of information therein; and

an oblivion processing unit for deleting information which has been stored for a predetermined period of time since the information was initially stored, among the pieces of information concerning states of emotion stored in said emotion information storing unit in the past, and for excluding at least information showing a larger amount of change in emotion than a predetermined amount and information matching a predetermined change pattern, from said information to be deleted.

6. The emotion detecting system according to claim 5, further comprising:

a sentence recognition unit for executing grammar analysis by processing information concerning any of the voice uttered by the subject and characters inputted by the subject, and for generating speech information expressing a meaning of a sentence; and

a storage controlling unit for storing the speech information generated by said sentence recognition unit in the emotion information storing unit, in synchronous with the information concerning said states of emotion.

7. The emotion detecting system according to claim 2, further comprising:

a voiceless time determining unit for determining a reference voiceless time based on a state of emotion among the detected states of emotion; and

a sentence segmentation detecting unit for detecting a segmentation of sentence of the voice by utilizing the reference voiceless time determined by said voiceless time determining unit.

8. Software including an emotion detecting program executable by a computer for detecting an emotion of a subject, wherein said emotion detecting program includes:

a step of inputting a voice signal;

a step of detecting an intensity of a voice, a tempo expressing speed the voice emerges at, respectively, and detecting, as a time value, intonation expressing an intensity-change pattern in each word the voice makes, based on the inputted voice signal;

a step of obtaining a first amount of change indicating a change in the intensity of the detected voice, in a direction of time axis, a second amount of change indicating a change in the tempo of the voice, in the direction of time axis, and a third amount of change indicating a change in the intonation in the voice, in the direction of time axis, respectively; and

a step of generating signals expressing states of emotion of at least anger, sadness, and pleasure, respectively, based on the obtained first, second, and third amounts of change.

9. A sensibility generating method, comprising the steps of:

retaining beforehand pieces of individuality information determining at least reason, a predetermined characteristic, and will of a subject that generates sensibility;

generating instinctive motivation information including at least a first instinct parameter expressing a degree of pleasure, a second instinct parameter expressing a degree of danger, and a third instinct parameter expressing a degree of achievement and change, based on an inputted situation information which indicates a state of any of a partner'"'"'s emotion and an environment the partner is in;

generating emotion information including a basic emotion parameter of at least pleasure, anger, and sadness, based on said instinctive motivation information generated; and

controlling said emotion information generated based on said individuality information.

10. A sensibility generator, comprising:

an instinct determining unit for inputting episode situation information indicating a partner'"'"'s emotion, an environment the partner is in, and the partner'"'"'s will, and for generating instinctive motivation information which indicates at least a first instinct parameter expressing a degree of pleasure, a second instinct parameter expressing a degree of danger, and a third instinct parameter expressing one of a degree of achievement and a degree of change, based on said episode situation information;

an emotion generating unit for generating emotion information including basic emotion parameters of at least pleasure, anger, and sadness, based on the instinctive motivation information outputted from said instinct determining unit;

an individuality information providing unit for providing individuality information determining at least reason and will with sensibility of a subject that generates sensibility; and

an emotion controlling unit for controlling emotion information outputted from said emotion generating unit, based on the individuality information provided from said individuality information providing unit.

11. The sensibility generator according to claim 10, wherein said emotion generating unit includes:

a life rhythm generating unit for generating information expressing any of an environment changing periodically and a life rhythm of a living body; and

a voluntary emotion controlling unit for controlling voluntary emotion in said emotion generating unit, according to the information on the life rhythm outputted by said life rhythm generating unit.

12. The sensibility generator according to claim 10, wherein said emotion generating unit includes:

an instinct-to-emotion information retaining unit for retaining pattern information which allows said basic emotion parameter and said instinctive motivation information to correspond to each other; and

a matching probability learning unit for outputting information expressing a probability of matching/mismatching of the instinctive motivation information with the pattern information of said instinct-to-emotion information retaining unit, the instinctive motivation information being outputted from said instinct determining unit.

13. The sensibility generator according to claim 10, wherein said emotion generating unit includes:

an emotion feedback controlling unit for inputting to the emotion generating unit at least its own emotion information finally generated, and for reflecting the finally generated information on its own emotion information to be generated subsequently.

14. The sensibility generator according to claim 10, wherein said emotion controlling unit reflects information of a life rhythm, which is an individuality of a subject that generates sensibility, on the emotion information to be inputted.

15. The sensibility generator according to claim 10, further comprising:

a knowledge database for storing situation information showing a past situation, a past episode, and a result of the past situation and episode;

a knowledge collating unit for retrieving and extracting past situation information analogous to newly inputted situation information from said knowledge database, and for providing the past situation information to said emotion controlling unit; and

a data update controlling unit for updating contents of said knowledge database based on the situation information showing a newly inputted situation and a result of the new situation, and for automatically deleting, from said knowledge database, situation information of low priority in the order of time in accordance with weight of the contents.

16. The sensibility generator according to claim 10, further comprising:

a voice inputting unit for inputting a voice signal;

an intensity detecting unit for detecting an intensity of a voice based on the voice signal inputted by said voice inputting unit;

a tempo detecting unit for detecting speed the voice emerges at as a tempo based on the voice signal inputted by said voice inputting unit;

an intonation detecting unit for detecting intonation expressing an intensity-change pattern in a word of the voice based on the voice signal inputted by said voice inputting unit;

a change-amount detecting unit for obtaining amounts of change in the intensity of the voice detected by said intensity detecting unit, the tempo of the voice detected by said tempo detecting unit, and the intonation in the voice detected by said intonation detecting unit, respectively; and

an emotion detecting unit for outputting signals expressing states of emotion of at least anger, sadness, and pleasure, respectively, based on the amounts of change detected by said change-amount detecting unit.

17. The sensibility generator according to claim 16, further comprising:

a voice recognition unit for recognizing the voice inputted from said voice inputting unit, and for outputting character information; and

a natural language processing unit for subjecting vocal information recognized by said voice recognition unit to natural language processing, and for generating meaning information expressing a meaning of the inputted voice.

18. Software including a program and data executable by a computer utilized for sensibility generation control, wherein said program includes:

a step of generating instinctive motivation information including at least a first instinct parameter expressing a degree of pleasure, a second instinct parameter expressing a degree of danger, and a third instinct parameter expressing any of a degree of achievement and change, based on an inputted situation information which indicates a state of any of a partner'"'"'s emotion and an environment the partner is in;

a step of generating emotion information including a basic emotion parameter of at least pleasure, anger, and sadness, based on said instinctive motivation information generated;

a step of providing individuality information determining at least reason and will of a subject that generates sensibility; and

a step of controlling said emotion information generated, based on said individuality information.

19. An intonation detecting method for detecting, from a voice signal, an intonation used for detecting an emotion of a subject, comprising the steps of:

inputting a voice signal;

detecting areas having same frequency components based on an intensity-change pattern in a word expressed by the inputted voice signal; and

detecting time intervals at which each of said areas having the same frequency components appears, to utilize the time intervals as the intonation.

20. An intonation detecting system for detecting, from a voice signal, an intonation used for detecting an emotion of a subject, comprising:

a voice inputting unit for inputting a voice signal;

an area detecting unit for detecting areas having same frequency components based on an intensity-change pattern in a word expressed by the inputted voice signal; and

an intonation detecting unit for detecting time intervals at which each of said areas having the same frequency components appears, to utilize the time intervals as the intonation.

21. A recording medium including a program executable by a computer for detecting, from a voice signal, an intonation used for detecting an emotion of a subject, comprising:

a step of inputting a voice signal;

a step of detecting areas having same frequency components based on an intensity-change pattern in a word of the inputted voice signal; and

a step of detecting time intervals at which each of said areas having the same frequency components appears, to utilize the time intervals as the intonation.