Method and apparatus for synthesising an emotion conveyed on a sound

• US 20030093280A1
• Filed: 07/11/2002
• Published: 05/15/2003
• Est. Priority Date: 07/13/2001
• Status: Abandoned Application

• Current Assignee: Sony France SA (Sony Group Corp.)
• Original Assignee: Sony France SA (Sony Group Corp.)
• Inventors: Oudeyer, Pierre-Yves
• Application Number:
  US10/192,974
• Publication Number:
  US 20030093280A1
• Time in Patent Office: Days
• US Class Current: 704/266
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

An emotion conveyed on a sound is synthesised by selectively modifying elementary sound portions (S) thereof prior to delivering the sound through an operator application step (S10, S16, S20) in which at least one operator (OP, OD; OI) is selectively applied to the elementary sound portions (S) to impose a specific modification in a characteristic, such as pitch and/or duration in accordance with an emotion to be synthesised.

This can be achieved by applying at least one operator (OIrs, OIfs, OIsu, OIsd) to modify an intensity characteristic of an elementary sound portion, and/or an operator (OPrs, OPfs) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall according to an imposed slope characteristic (Prs, Pfs), and/or an operator (OPsu, OPsd) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall uniformly by a determined value (Psu, Psd), and/or an operator (ODd, ODc) for selectively causing the duration (t1) of an elementary sound portion (S) to increase or decrease by a determined value (D).

Claims

1. Method of synthesising an emotion conveyed on a sound, by selectively modifying at least one elementary sound portion (S) thereof prior to delivering the sound,

characterised in that said modification is produced by an operator application step (S10, S16, S20) in which at least one operator (OP, OD;

OI) is selectively applied to at least one said elementary sound portion (S) to impose a specific modification in a characteristic thereof in accordance with an emotion to be synthesised.

2. Method according to claim 1, wherein said characteristic comprises at least one of:

pitch, and

duration

of said elementary sound portions (S).

3. Method according to claim 2, wherein said operator application step (S10, S16, S20) comprises forming at least one set of operators (OS(U), OS(PA), OS(FL)), said set comprising at least one operator (OPrs, OPfs, OPsu, OPsd) to modify a pitch characteristic and/or at least one operator (ODd, ODc) to modify a duration characteristic of said elementary sound portions (S).

4. Method according to any one of claims 1 to 3, wherein said operator application step (S10, S16, S20) comprises applying at least one operator (OIrs, OIfs, OIsu, OIsd) to modify an intensity characteristic of said elementary sound portions.

5. Method according to any one of claims 1 to 4, further comprising, a step (S8, S12, S18) of parameterising at least one said operator (OP, OI, OD) with a numerical parameter affecting an amount of a said specific modification associated to said operator in accordance with an emotion to be synthesised.

6. Method according to any one of claims 1 to 5, wherein said operator application step (S10, S16, S20) comprises applying an operator (OPrs, OPfs) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall according to an imposed slope characteristic (Prs, Pfs).

7. Method according to any one of claims 1 to 6, wherein said operation application step (S10, S16, S20) comprises applying an operator (OPsu, OPsd) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall uniformly by a determined value (Psu, Psd).

8. Method according to any one of claims 1 to 7, wherein said operation application step (S10, S16, S20) comprises applying an operator (ODd, ODc) for selectively causing the duration (t1) of an elementary sound portion (S) to increase or decrease by a determined value (D).

9. Method according to any one of claims 1 to 8, comprising a universal phase (P2) in which at least one said operator (OP(U), OD(U)) is applied (S10) systematically to all elementary sound portions (S) forming a determined sequence of said sound.

10. Method according to claim 9, wherein said at least one operator (OP(U), OD(U)) is applied with a same operator parameterisation (S8) to all elementary sound portions (S) forming a determined sequence of said sound.

11. Method according to any one of claims 1 to 10, comprising a probabilistic accentuation phase (P3) in which at least one said operator (OP(PA), OD(PA)) is applied (S16) only to selected elementary sound portions (S) chosen to be accentuated.

12. Method according to claim 11, wherein said selected elementary sound portions (S) are selected by a random draw (S14) from candidate elementary sound portions (S).

13. Method according to claim 12, wherein said random draw selects elementary sound portions (S) with a probability (N) which is programmable.

14. Method according to claim 12 or 13, wherein said candidate elementary sound portions are:

all elementary sound portions when a source (10) of said portions does not prohibit an accentuation on some data portions, or

only those elementary sound portions that are not prohibited from accentuation when said source (10) prohibits accentuations on some data portions.

15. Method according to any one claims 11 to 14 wherein a same operator parameterisation (S12) is used for said at least one operator (OP(PA), OD(PA) applied in said probabilistic accentuation phase (P3).

16. Method according to any one of claims 1 to 15, comprising a first and last elementary sound portions accentuation phase (S4) in which at least one said operator (OP(FL), OD(FL)) is applied (S10) only to a group of at least one elementary sound portion forming the start and end of said determined sequence of sound.

17. Method according to any one of claims 9 to 16, wherein said determined sequence of sound is a phrase.

18. Method according to any one of claims 1 to 17, wherein said elementary portions of sound (S) corresponds to a syllable or to a phoneme.

19. Method according to any one of claims 1 to 18, wherein said elementary sound portions correspond to intelligible speech.

20. Method according to any one of claims 1 to 19, wherein said elementary sound portions correspond to unintelligible sounds.

21. Method according to any one of claims 1 to 20, wherein said elementary sound portions are presented as formatted data values specifying a duration (t1) and/or at least one pitch value (P1-P5) existing over determined parts of or all said duration of said elementary sound.

22. Method according to claim 20, wherein said operators (OP, OP, OD) act to selectively modify said data values.

23. Method according to claim 21 or 22, performed without changing the data format of said elementary sound portion data and upstream of an interpolation stage (14), whereby said interpolation stage can process data modified in accordance with an emotion to be synthesised in the same manner as for data obtained from an arbitrary source (10) of elementary sound portions (S).

32. A data medium comprising software module means for executing the method according to any one of claims 1 to 23.

24. Device for synthesising an emotion conveyed on a sound, using means for selectively modifying at least one elementary sound portion (S) thereof prior to delivering the sound,

characterised in that said means comprise operator application means (22) for applying (S10, S16, S20) at least one operator (OP, OD;

OI) to at least one said elementary sound portion (S) to impose a specific modification in a characteristic thereof in accordance with an emotion to be synthesised.

25. Device according to claim 24, wherein said operator application means (22) comprises means (26, 28) for forming at least one set of operators (OS(U), OS(PA), OS(FL)), said set comprising at least one operator (OPrs, OPfs, OPsu, OPsd) to modify a pitch characteristic and/or at least one operator (ODd, ODc) to modify a duration characteristic of said elementary sound portions (S).

26. Device according to any one of claims 24 or 25, comprising an operator (OPrs, OPfs) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall according to an imposed slope characteristic (Prs, Pfs).

27. Device according to any one of claims 24 to 26, comprising an operator (OPsu, OPsd) for selectively causing the time evolution of the pitch of an elementary sound portion (S) to rise or fall uniformly by a determined value (Psu, Psd).

28. Device according to any one of claims 24 to 27, comprising an operator (ODd, ODc) for selectively causing the duration (t1) of an elementary sound portion (S) to increase or decrease by a determined value (D).

29. Device according to any one of claims 24 to 28, operative to conduct at least one of the following three stages:

i) a universal phase (P2) in which at least one said operator (OP(U), OD(U)) is applied (S10) systematically to all elementary sound portions (S) forming a determined sequence of said sound;

ii) a probabilistic accentuation phase (P3) in which at least one said operator (OP(PA), OD(PA)) is applied (S16) only to selected elementary sound portions (S) chosen to be accentuated; and

iii) a first and last elementary sound portions accentuation phase (S4) in which at least one said operator (OP(FL), OD(FL)) is applied (S10) only to a group of at least one elementary sound portion forming the start and end of said determined sequence of sound.

30. Device according to any one of claims 24 to 29, wherein said operator application means (22) operate on externally supplied formatted data values specifying a duration (t1) and/or at least one pitch value (P1-P5) existing over determined parts of or all said duration of said elementary sound.

31. Device according to claim 30, wherein said operator application means (22) operate without changing the data format of said elementary sound portion data and upstream of an interpolation stage (14), whereby said interpolation stage can process data modified in accordance with an emotion to be synthesised in the same manner as for data obtained from an arbitrary source (10) of elementary sound portions (S).