Voice synthesizing method and voice synthesizing apparatus

• US 10,002,604 B2
• Filed: 11/14/2013
• Issued: 06/19/2018
• Est. Priority Date: 11/14/2012
• Status: Active Grant

• Current Assignee: Yamaha Corporation
• Original Assignee: Yamaha Corporation
• Inventors: Kayama, Hiraku, Nishitani, Yoshiki
• Primary Examiner(s): Sanchez, Michael Ortiz
• Application Number:
  US14/080,660
• Publication Number:
  US 20140136207A1
• Time in Patent Office: 1,678 Days
• Field of Search: 704258-269, 704272, 704278, 84653, 84678, 84744
• CPC Class Codes: G10H 1/02 Means for controlling the tone frequencies, e.g. attack or decay; Means for producing special musical effects, e.g. vibratos or glissandos
• CPC Class Codes: G10H 1/0551 using variable capacitors
• CPC Class Codes: G10H 1/344 Structural association with individual keys electrically operated wind-actuated organs G10B3/22
• CPC Class Codes: G10H 2220/271 Velocity sensing for individual keys, e.g. by placing sensors at different points along the kinematic path for individual key velocity estimation by delay measurement between adjacent sensor signals velocity sensing common to several keys G10H2220/221
• CPC Class Codes: G10H 2230/135 Spint guitar, i.e. guitar-like instruments in which the sound is not generated by vibrating strings, e.g. guitar-shaped game interfaces
• CPC Class Codes: G10H 2230/155 Spint wind instrument, i.e. mimicking musical wind instrument features; Electrophonic aspects of acoustic wind instruments; MIDI-like control therefor. wind instrument sound synthesis G10H2250/461; mouth control, e.g. breath G10H2220/361; natural aerodynamic noise synthesis, e.g. wind G10H2250/431
• CPC Class Codes: G10H 2230/251 Spint percussion, i.e. mimicking percussion instruments; Electrophonic musical instruments with percussion instrument features; Electrophonic aspects of acoustic percussion instruments, MIDI-like control therefor gensound percussion G10H2250/435
• CPC Class Codes: G10H 2240/311 MIDI transmission G10H2240/056 takes precedence
• CPC Class Codes: G10H 2250/455 Gensound singing voices, i.e. generation of human voices for musical applications, vocal singing sounds or intelligible words at a desired pitch or with desired vocal effects, e.g. by phoneme synthesis formant synthesis G10H2250/481; parcor synthesis G10H2250/505; modulation effects G10H2210/195; ensemble effects G10H2210/245; speech synthesis in general G10L13/00
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination

Abstract

A voice synthesizing apparatus includes a first receiver configured to receive first utterance control information generated by detecting a start of a manipulation on a manipulating member by a user, a first synthesizer configured to synthesize, in response to a reception of the first utterance control information, a first voice corresponding to a first phoneme in a phoneme sequence of a voice to be synthesized to output the first voice, a second receiver configured to receive second utterance control information generated by detecting a completion of the manipulation on the manipulating member or a manipulation on a different manipulating member, and a second synthesizer configured to synthesize, in response to a reception of the second utterance control information, a second voice including at least the first phoneme and a succeeding phoneme being subsequent to the first phoneme of the voice to be synthesized to output the second voice.

Claims

1. A voice synthesizing method comprising:

a first receiving step of receiving first utterance control information generated on detecting a start of a manipulation on an input device by a user, wherein the start of the manipulation is an initial interaction with the input device by the user;

a first synthesizing step of synthesizing and outputting, in accordance with a timing of the reception of the first utterance control information, a first voice including a first phoneme in a phoneme sequence of a voice to be synthesized;

a second receiving step of receiving second utterance control information generated on detecting a completion of the manipulation on the input device by the user, wherein the completion of the manipulation is a completion of the initial interaction with the input device by the user; and

a second synthesizing step of synthesizing and outputting, in accordance with a timing of the reception of the second utterance control information, a second voice including a succeeding phoneme in the phoneme sequence, the succeeding phoneme being subsequent to the first phoneme in the phoneme sequence.

2. The voice synthesizing method according to claim 1, wherein the first voice comprises a part of transition from a silence or a preceding phoneme to the first phoneme and the first phoneme;

and

wherein the second voice comprises at least a part of transition from the first phoneme to the succeeding phoneme and the succeeding phoneme.

3. The voice synthesizing method according to claim 1, wherein the first synthesizing step and the second synthesizing step are performed with using synthesis information including phoneme sequence information representative of the phoneme sequence of the voice to be synthesized and pitch information representative of a pitch;

wherein the user manipulates the input device to provide an instruction to start utterance of the first voice synthesized with using the synthesis information, and through the manipulation the user can specify the pitch of the first voice;

wherein the first utterance control information includes the pitch information representing the pitch specified through the manipulation on the input device; and

wherein in the first synthesizing step, the first voice is synthesized with using the pitch information included in the first utterance control information.

4. The voice synthesizing method according to claim 3, wherein when successively receiving a plurality of pieces of first utterance control information each including pitch information representative of a different pitch, the first voice is synthesized with using the pitch information included in one piece selected from among the plurality of pieces of first utterance control information.

5. The voice synthesizing method according to claim 3, wherein when successively receiving a plurality of pieces of second utterance control information each including information representative of a different velocity or volume, the second voice is synthesized with using the information included in one piece selected from among the plurality of pieces of second utterance control information.

6. The voice synthesizing method according to claim 3, wherein when receiving a plurality of utterance control information pairs, each of the pairs formed of the first and the second utterance control information including pitch information representative of the same pitch, each pair of the utterance control information pairs includes the pitch information representative of a different pitch from the other pairs, voice synthesis is simultaneously performed for each utterance control information pair in parallel.

7. The voice synthesizing method according to claim 1, further comprising:

outputting third utterance control information to provide an instruction to stop an output of the first voice when the reception of the second utterance control information is not detected within a predetermined time from the output of the first utterance control information.

8. The voice synthesizing method according to claim 4, wherein the first voice is synthesized with using the pitch information included in an earliest received one piece selected from among the plurality of pieces of first utterance control information.

9. The voice synthesizing method according to claim 4, wherein the first voice is synthesized with using the pitch information included in a last received one piece selected from among the plurality of pieces of first utterance control information.

10. The voice synthesizing method according to claim 1, further comprising:

a third receiving step of receiving third utterance control information generated on detecting a completion of the manipulation on the input device by the user, wherein the third utterance control information includes pitch information and a velocity or a volume;

a third synthesizing step of synthesizing and outputting, in accordance with a timing of the reception of the third utterance control information, a third voice; and

a switching step of switching between a first operation mode and a second operation mode,

wherein in the first operation mode, the first receiving step, the first synthesizing step, the second receiving step, and the second synthesizing step are performed; and

wherein in the second operation mode, the third receiving step and the second synthesizing step are performed.

11. The voice synthesizing method according to claim 1, wherein the detection of the start of the manipulation on the input device by the user includes a detection of the user'"'"'s finger approaching the input device.

15. The voice synthesizing method according to claim 1,

wherein when the first phoneme represents a consonant:

the first voice is synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the second voice is synthesized and output in accordance with the timing of the reception of the second utterance control information; and

wherein when the phoneme sequence represents one vowel;

the voice including the phoneme sequence is synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the voice including the phoneme sequence is not synthesized and output in accordance with the timing of the reception of the second utterance control information.

16. The voice synthesizing method according to claim 1,

wherein when the first phoneme represents a consonant:

the first voice is synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the second voice is synthesized and output in accordance with the timing of the reception of the second utterance control information; and

wherein when the phoneme sequence represents one vowel;

the voice including the phoneme sequence is not synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the voice including the phoneme sequence is synthesized and output in accordance with the timing of the reception of the second utterance control information.

19. The voice synthesizing method according to claim 1,

wherein the first phoneme represents a consonant, and

the succeeding phoneme represents a vowel or a transition from the consonant to a vowel.

12. A voice synthesizing apparatus comprising:

a first receiver configured to receive first utterance control information generated on detecting a start of a manipulation on an input device by a user, wherein the start of the manipulation is an initial interaction with the input device by the user;

a first synthesizer configured to synthesize and output, in accordance with a timing of the reception of the first utterance control information, a first voice including a first phoneme in a phoneme sequence of a voice;

a second receiver configured to receive second utterance control information generated on detecting a completion of the manipulation on the input device by the user, wherein the completion of the manipulation is a completion of the initial interaction with the input device by the user; and

a second synthesizer configured to synthesize and output, in accordance with a timing of the reception of the second utterance control information, a second voice including a succeeding phoneme in the phoneme sequence, the succeeding phoneme being subsequent to the first phoneme in the phoneme sequence.

13. The voice synthesizing apparatus according to claim 12, further comprising:

a first sensor configured to detect the start of the manipulation on the input device by the user; and

a second sensor configured to detect the completion of the manipulation on the input device.

14. The voice synthesizing apparatus according to claim 12, wherein the second receiver receives the second utterance control information generated on only detecting the completion of the manipulation on the input device by the user.

17. The voice synthesizing apparatus according to claim 12,

wherein when the first phoneme represents a consonant:

the first voice is synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the second voice is synthesized and output in accordance with the timing of the reception of the second utterance control information; and

wherein when the phoneme sequence represents one vowel;

the voice including the phoneme sequence is synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the voice including the phoneme sequence is not synthesized and output in accordance with the timing of the reception of the second utterance control information.

18. The voice synthesizing apparatus according to claim 12,

wherein when the first phoneme represents a consonant:

the first voice is synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the second voice is synthesized and output in accordance with the timing of the reception of the second utterance control information; and

wherein when the phoneme sequence represents one vowel;

the voice including the phoneme sequence is not synthesized and output in accordance with the timing of the reception of the first utterance control information, and

the voice including the phoneme sequence is synthesized and output in accordance with the timing of the reception of the second utterance control information.

20. The voice synthesizing apparatus according to claim 12,

wherein the first phoneme represents a consonant, and

the succeeding phoneme represents a vowel or a transition from the consonant to a vowel.

21. An operating apparatus comprising:

a plurality of input devices each configured to receive a manipulation by a user;

a first sensor configured to detect a start of the manipulation conducted to one of the plurality of input devices by the user, wherein the start of the manipulation is an initial interaction with the one of the plurality of input devices by the user;

a second sensor configured to;

detect a completion of the manipulation conducted to the one of the plurality of input devices by the user, wherein the completion of the manipulation is a completion of the initial interaction with the one of the plurality of input devices by the user, and

generate second utterance control information at a timing of the detection of the start of the manipulation;

a first generator configured to output first utterance control information at the timing of the detection of the start of the manipulation, the first utterance control information used for instructing a voice synthesizing apparatus to synthesize and output, in accordance with a timing of a reception of the first utterance control information, a first voice including a first phoneme in a phoneme sequence of a voice to be synthesized; and

a second generator configured to output the second utterance control information at a timing of the detection of the completion of the manipulation, the second utterance control information used for instructing the voice synthesizing apparatus to synthesize and output, in accordance with a timing of a reception of the second utterance control information, a second voice including a succeeding phoneme in the phoneme sequence, and the succeeding phoneme being subsequent to the first phoneme in the phoneme sequence.