Emotion recognition method and device

• US 20030055654A1
• Filed: 07/12/2002
• Published: 03/20/2003
• Est. Priority Date: 07/13/2001
• Status: Active Grant

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Oudeyer, Pierre Yves
• Granted Patent: US 7,451,079 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

The emotion recognition is performed by:

extracting (70) a set comprising at least one feature derived from said signal, and

processing (84, 90) the set of extracted feature(s) to detect an emotion therefrom.

There is further provided a step of low-pass filtering (74) the voice signal prior to extracting therefrom at least one feature of the set. The cut-off frequency for the low pass filtering is typically centred around 250 Hz.

The features are e.g. statistical quantities extracted from sampling a signal of the intensity or pitch of the voice signal.

Claims

1. A method of detecting an emotion conveyed in a voice signal, comprising the steps of:

extracting (70) a set comprising at least one feature derived from said signal, and

processing (84, 90) said set of extracted feature(s) to detect an emotion therefrom,

characterised in that it comprises a step of low-pass filtering (74) said voice signal prior to extracting therefrom at least one said feature of said set.

2. Method according to claim 1, wherein said low-pass filtering step comprises filtering (74) the intensity of said voice signal with a cut-off frequency (Fco) substantially in the range of 150 to 400 Hz.

3. Method according to claim 2, wherein said cut-off frequency (Fco) is substantially in the range of 250 to 300 Hz.

4. Method according to claim 3, wherein said cut-off frequency (Fco) is substantially equal to 250 Hz.

5. Method according to any one of claims 1 to 4, wherein the feature extracting step comprises the steps of selectively extracting one or more among 160 features obtained from the following combination of parameters I, II, and II:

Parameters I;

i) the intensity of the signal derived after said low-pass filtering step (74),

ii) the intensity of the signal without low-pass filtering,

iii) the pitch of the signal without low-pass filtering, and

iv) the pitch of the signal derived after said low-pass filtering step (74);

Parameters II, from a sequence of values of a parameter I obtained over a duration of said voice signal;

1) a series of local minimas, whose members are values in said sequence that correspond to a minima,

2) a series of local maximas, whose members are values in said sequence that correspond to a maxima,

3) a series of durations (time intervals), whose members are time values separating local (adjacent) extremas, and

4) a series, whose members are all the successive sampled values of the sequence; and

Parameters III, calculated from a series of parameter II;

1) the mean,

2) the maximum (i.e. the overall highest value),

3) the minimum (i.e. the overall lowest vale),

4) the difference between the maximum and the minimum,

5) the variance,

6) the median,

7) the first quartile,

8) the third quartile,

9) the interquartile range, defined as the difference between the third and first quartiles,

10) the mean of the absolute value of the local derivative;

at least one of said features originating from;

i) the intensity of the signal after said low-pass filtering step (74) or from iv) the pitch of the signal after low-pass filtering step (74) of parameters I.

6. Method according to claim 5, wherein said parameters I comprise:

i) the intensity of the signal derived after said low-pass filtering step (74),

ii) the intensity of the signal without low-pass filtering, and

iii) the pitch of the signal without low-pass filtering,

whereby said combination yields a total of 120 different possible features;

at least one of said features originating from;

i) the intensity of the signal after said low-pass filtering step (74) of parameters I.

7. Method according to claim 5 or 6, wherein said parameters III comprise the following five parameters:

2) the maximum (i.e. the overall highest value),

3) the minimum (i.e. the overall lowest vale),

6) the median,

7) the first quartile,

8) the third quartile,

whereby said combination yields a total of 80 different possible features with claim 5, and 50 different possible combinations with claim 6.

8. Method according to claim 7, wherein said extracting step comprises extracting at least one of the following possible features:

the maximum (i.e. the overall highest value),

the minimum (i.e. the overall lowest vale),

the median,

the first quartile,

the third quartile,

calculated from;

the series of values of the intensity signal derived after said low-pass filtering of the voice signal,

the series of minimum values of the intensity signal derived without low-pass filtering the voice signal,

the series of pitch values of the signal derived after said low-pass filtering of the voice signal, and

the series of pitch values of the signal derived without low-pass filtering of the voice signal.

9. Method according to any one of claims 1 to 8, wherein said extracting step comprises extracting features derived from both said voice signal after said low low-pass filtering step (74) and from said voice signal without low-pass filtering.

10. Method according to any one of claims 1 to 8, wherein said extracting step comprises extracting features only from said voice signal after said low-pass filtering step (74).

11. Method according to any one of claims 1 to 10, wherein said at least one member of said set of features comprises:

1;

the median value of the signal intensity values derived after low-pass filtering the voice signal,

2;

the mean value of the signal intensity values derived after low-pass filtering the voice signal,

3;

third quartile value of the signal intensity values derived after low-pass filtering the voice signal,

4;

the first quartile value of the signal intensity values derived after low-pass filtering the voice signal,

5;

the maximum of the signal intensity values derived after low-pass filtering the voice signal, and

6;

the maximum value of the series of the signal intensity values derived after low-pass filtering the voice signal.

12. Method according to any one of claims 1 to 11, further comprising a step of smoothing a signal extracted from said voice signal prior to said extracting step.

13. Method according to any one of claims 1 to 12, further comprising a step of generating an emotion detecting algorithm using a teaching algorithm, said teaching algorithm exploiting said at least one feature extracted from a low-pass filtered voice signal.

14. Method according to claim 13, further comprising a step of programming (PU) an emotion detection device (60) with an off-line emotion detection algorithm (84) and/or an on-line emotion detection algorithm (90), whereby said algorithms exploit at least on feature extracted from a low-pass filtered voice signal.

15. A device for detecting an emotion conveyed in a voice signal, comprising the:

extracting means (70) for extracting a set comprising at least one feature derived from said signal, and

processing means (84, 90) for processing said set of extracted feature(s) to detect an emotion therefrom,

characterised in that it comprises a low-pass filtering means (74) for low-pass filtering said voice signal prior to extracting therefrom at least one said feature of said set.

16. Device according to claim 15, wherein said low-pass filtering means (74) has a cut-off frequency (Fco) substantially in the range of 150 to 400 Hz.

17. Device according to claim 16, wherein said cut-off frequency (Fco) is substantially in the range of 250 to 300 Hz.

18. Device according to claim 17, wherein said cut-off frequency (Fco) is substantially equal to 250 Hz.

19. Device according to any one of claims 15 to 18, wherein the feature extracting means (70) comprises means for selectively extracting one or more among 160 features obtained from the following combination of parameters I, II, and II:

Parameters I;

i) the intensity of the signal derived after said low-pass filtering step (74),

ii) the intensity of the signal without low-pass filtering,

iii) the pitch of the signal without low-pass filtering, and

iv) the pitch of the signal derived after said low-pass filtering step (74);

Parameters II, from a sequence of values of a parameter I obtained over a duration of said voice signal;

1) a series of local minimas, whose members are values in said sequence that correspond to a minima,

2) a series of local maximas, whose members are values in said sequence that correspond to a maxima,

3) a series of durations (time intervals), whose members are time values separating local (adjacent) extremas, and

4) a series, whose members are all the successive sampled values of the sequence; and

Parameters III, calculated from a series of parameter II;

1) the mean,

2) the maximum (i.e. the overall highest value),

3) the minimum (i.e. the overall lowest vale),

4) the difference between the maximum and the minimum,

5) the variance,

6) the median,

7) the first quartile,

8) the third quartile,

9) the interquartile range, defined as the difference between the third and first quartiles,

10) the mean of the absolute value of the local derivative;

at least one of said features originating from;

i) the intensity of the signal after said low-pass filtering step (74) or from iv) the pitch of the signal after low-pass filtering step (74) of parameters I.

20. Device according to any one of claims 15 to 19, wherein said extracting means (70) is set to extract features derived from both said voice signal after said low low-pass filtering means (74) and from said voice signal without low-pass filtering.

21. Device according to any one of claims 15 to 19, wherein said extracting means (70) are set to extract features only from said voice signal after said low-pass filtering means (74).

22. Device according to any one of claims 15 to 21, wherein said at least one member of said set of features comprises:

1;

the median value of the signal intensity values derived after low-pass filtering the voice signal,

2;

the mean value of the signal intensity values derived after low-pass filtering the voice signal,

3;

third quartile value of the signal intensity values derived after low-pass filtering the voice signal,

4;

the first quartile value of the signal intensity values derived after low-pass filtering the voice signal,

5;

the maximum of the signal intensity values derived after low-pass filtering the voice signal, and

6;

the maximum value of the series of the signal intensity values derived after low-pass filtering the voice signal.

23. Device according to any one of claims 15 to 22, further comprising means for smoothing a signal extracted from said voice signal prior to said extracting means (70).

24. Device according to any one of claims 15 to 23, wherein said processing means comprises an off-line emotion recognition algorithm unit (84) containing a preprogrammed algorithm.

25. Device according to any one of claims 15 to 24, wherein said processing means comprises an on-line emotion recognition algorithm unit (90), said device further comprising a teaching algorithm unit (88) for producing an emotion recognition algorithm in response to data produced said extracting means (70).

26. A voice signal processing device specifically for the device according to any one of claims 15 to 25, characterised in that it comprises low-pass filtering means (74) for supplying a low-passed form of said voice signal to said extracting means (70).