METHODS FOR ALIGNING EXPRESSIVE SPEECH UTTERANCES WITH TEXT AND SYSTEMS THEREFOR

• US 20130262096A1
• Filed: 09/21/2012
• Published: 10/03/2013
• Est. Priority Date: 09/23/2011
• Status: Active Grant

• Current Assignee: Lessac Technologies, Inc.
• Original Assignee: Lessac Technologies, Inc.
• Inventors: WILHELMS-TRICARICO, Reiner, MOTTERSHEAD, Brian, REICHENBACH, John B., NITISAROJ, Rattima, BAUMGARTNER, Michael, MARPLE, Gary A.
• Granted Patent: US 10,453,479 B2
• Time in Patent Office: Days
• US Class Current: 704/202
• CPC Class Codes: G10L 13/04 Details of speech synthesis systems, e.g. synthesiser structure or memory management
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 15/1807 using prosody or stress
• CPC Class Codes: G10L 25/30 using neural networks
• CPC Class Codes: G10L 25/75 for modelling vocal tract parameters
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

A system-effected method for synthesizing speech, or recognizing speech including a sequence of expressive speech utterances. The method can be computer-implemented and can include system-generating a speech signal embodying the sequence of expressive speech utterances. Other possible steps include: system-marking the speech signal with a pitch marker indicating a pitch change at or near a first zero amplitude crossing point of the speech signal following a glottal closure point, at a minimum, at a maximum or at another location; system marking the speech signal with at least one further pitch marker; system-aligning a sequence of prosodically marked text with the pitch-marked speech signal according to the pitch markers; and system outputting the aligned text or the aligned speech signal, respectively. Computerized systems, and stored programs for implementing method embodiments of the invention are also disclosed.

Claims

1. A system-effected method for aligning a sequence of expressive speech utterances with corresponding text, the method being implemented on a computerized system and comprising:

system-processing a speech signal embodying the sequence of expressive speech utterances;

system-marking the speech signal with a pitch marker at a predetermined point in a cycle of the speech signal, optionally, at or near a first zero amplitude crossing point of the speech signal following a glottal closure point, the pitch marker indicating a pitch change in the speech signal;

system marking the speech signal with at least one further pitch marker at the same predetermined point in a further cycle of the speech signal; and

system-aligning a sequence of prosodically marked text corresponding with the pitch-marked speech signal according to the pitch markers in the speech signal.

2. A method according to claim 1 wherein the pitch markers correspond with pitch change indications or pitch change implications in the prosodically marked text.

3. A method according to claim 1 comprising determining a glottal closure point by estimating vocal-tract-related filter properties of a portion of the speech signal and applying inverse filtering to the speech signal, using the estimated filter properties.

4. A method according to claim 3 comprising:

generating an error signal from the filtered speech signal, the error signal resembling the vocal glottal excitation signal; and

identifying the glottal closure point as a point where the error signal rapidly drops after ramping up.

5. A method according to claim 1 comprising continuously determining glottal closure points and marking the signal with pitch markers for a zero amplitude crossing points of the speech signal at or near each determined glottal closure point.

6. A method for synthesizing speech or for recognizing speech, comprising a method according to claim 1 and system-outputting the aligned text or the aligned speech signal, respectively.

9. A method according to claim 6 comprising feeding back the acoustic data predicted for at least one previous phonetic subunit as input to the system learning model, optionally, for from one to ten previous phonetic subunits as input to the system learning model.

10. A method according to claim 9 comprising repeatedly processing at least one sentence from a trained voice database to reduce error variance.

11. A method according to claim 6 comprising sequencing the acoustic data according to prosodic variables wherein sequencing is system-effected by employing a text corpus pronounced by a speaker using at least two prosodies and an acoustic database for each prosody, wherein the acoustic database is aligned according to the different pitch, amplitude, and duration inflections and rhythms of a different set of prosodic rules for each prosody.

12. A method for synthesizing speech or for recognizing speech, comprising a method according to claim 6 and system-outputting the aligned text or the aligned speech signal, respectively.

19. A computerized system, optionally, being a speech synthesizer or a speech recognizer, comprising non-transitory computer-readable media and at least one program stored in the non-transitory computer-readable media, the program including instructions for performing a method according to claim 1.

20. Non-transitory computer media storing at least one program including instructions for performing a method according to claim 1.

7. A system-effected method for aligning a sequence of expressive speech utterances with corresponding text, the method being implemented on a computerized system and comprising:

system-processing a speech signal embodying the sequence of expressive speech utterances; and

system-aligning a sequence of prosodically marked text with the speech signal;

wherein the method includes;

system-encoding expressive intonational content derived from the prosodically marked text into each one of a sequence of phonetic subunits, each phonetic subunit bearing segmental information from the text;

inputting to a system learning model;

(i) acoustic data for each phonetic subunit in the phonetic subunit sequence, the phonetic subunit being addressed as a current phonetic subunit; and

(ii) for each current phonetic subunit, acoustic data for at least two phonetic subunits contextually related in the phonetic subunit sequence to the current phonetic subunit;

operating the system learning model to predict for the current phonetic subunit acoustic parameters to specify the current phonetic subunit, the prediction being made along multiple paths to yield multiple predictions;

selecting for the phonetic subunit one of the multiple acoustic parameter predictions for the current phonetic subunit;

including the selected acoustic parameters in the speech signal sequence.

8. A method according to claim 7 wherein the at least two phonetic subunits include the previous phonetic subunit to the current phonetic subunit and the two phonetic subunits next following the current phonetic subunit and, optionally,

include from one to ten previous phonetic subunits and from one to ten next phonetic subunits.

13. A system-effected method for aligning a sequence of expressive speech utterances with corresponding text, the method being implemented on a computerized system and comprising:

system-processing a speech signal embodying the sequence of expressive speech utterances, the speech utterances being pronounced according to defined prosodic rules; and

system-providing a sequence of prosodically marked text;

system-employing a neural network to relate the sequence of speech utterances to the sequence of prosodically marked text;

system-aligning a sequence of prosodically marked text with the speech signal.

14. A method according to claim 13 comprising operating the neural network recursively to reduce an error signal and enable the neural network to learn.

15. A method according to claim 13 wherein the neural network includes a reduced Boltzman machine and, optionally, a contrastive divergence learning module.

16. A method according to claim 13 comprising system-marking the speech signal with a pitch marker at a predetermined point in a cycle of the speech signal, optionally, at or near a first zero amplitude crossing point of the speech signal following a glottal closure point, the pitch marker indicating a pitch change in the speech signal and system marking the speech signal with at least one further pitch marker at the same predetermined point in a further cycle of the speech signal.

17. A method according to claim 16 wherein the pitch markers correspond with pitch change indications or pitch change implications in the prosodically marked text further comprising determining a glottal closure point by estimating vocal-tract-related filter properties of a portion of the speech signal and applying inverse filtering to the speech signal, using the estimated filter properties.

18. A method for synthesizing speech or for recognizing speech, comprising a method according to claim 13 and system-outputting the aligned text or the aligned speech signal, respectively.