Prosodic speech text codes and their use in computerized speech systems

• US 7,877,259 B2
• Filed: 03/07/2005
• Issued: 01/25/2011
• Est. Priority Date: 03/05/2004
• Status: Active Grant

• Current Assignee: Lessac Technologies, Inc.
• Original Assignee: Lessac Technologies, Inc.
• Inventors: Kur, Barry, Park, Sue Ann, Krebs, Nancy, Wilson, H. Donald, Gaary, Diane, Marple, Gary
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US11/547,369
• Publication Number:
  US 20070260461A1
• Time in Patent Office: 2,150 Days
• Field of Search: 704/260
• US Class Current: 704/260
• CPC Class Codes: G09B 5/04 with audible presentation of the material to be studied sound recording or reproducing G11B
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

A method of, and system for, acoustically coding text for use in the synthesis of speech from the text. The method includes marking the text to be spoken with one or more graphic symbols to indicate to a speaker a desired prosody to impart to the spoken text. The markups can include grapheme-phoneme pairs each wherein a visible prosodic-indicating grapheme is employed with written text and a corresponding digital phoneme is functional in the digital domain. The invention is useful in the generation of appealing, humanized machine speech for a wide range of applications, including voice mail systems, electronically enabled appliances, automobiles, computers, robotic assistants, games and the like, in spoken books and magazines, drama and other entertainment.

Claims

1. A computer-implemented method of synthesizing speech from text, the method comprising:

inputting text to be synthesized to a computerized system;

electronically marking the text with electronic versions of one or more prosodic graphical symbols, the electronically marked text being displayable or printable as human-readable text marked with the prosodic graphical symbols, to indicate to a speaker desired speech characteristics to be employed in speaking the text, wherein the prosodic graphical symbols indicate a desired prosody and include intelligibility pronunciation notations in sequence with the text and pitch change notations in sequence with the text; and

generating a synthesized speech output comprising phonetic data corresponding with the marked up text and having the prosody indicated by the prosodic graphical symbols.

2. A method according to claim 1 wherein the prosody comprises one or more prosody elements selected from the group consisting of pace, intonation pattern, rhythm, musicality, amplitude, pauses for emphasis and breath, and formal and informal articulations of words and phrases.

3. A method according to claim 2 comprising marking the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

graphical symbols indicating pitch control required for pronunciation of a letter, diphthong, syllable or other speech element according to a desired prosody;

an upglide to indicate rising pitch;

a downglide to indicate descending pitch;

a circumflex to indicate a rising then descending pitch or a descending then rising pitch;

a level sustention to indicate unchanged pitch;

a forward slash through a first letter followed by a closely related or identical sounding consonant to indicate that the first letter is prepared;

a shallow U-shaped liaison hammock to indicate that letters separated from each other by other letters are to be pronounced in a continuous manner, without pause between them;

a single underline to mark as playable percussives including tympani drumbeats, D, B and G and snare, bass and tom-tom drumbeats, T, P and K, respectively, unmarked consonants being not playable;

a double underline to mark as playable strings N, M, V and Z woodwinds, L, NG, TH and ZH and sound effects F, S, SH and th, unmarked consonants being not playable;

the letter combination “

hw”

marked over or adjacent to the letter combination WH in the text to be spoken to indicate that the H should be sounded first, followed by the W, and that neither should be played; and

Y and W connectives to indicate that speech continuity should be maintained from one word to the next or from one syllable to the next when a [Y] or a [W] occurs before another vowel, the Y and W connectives each comprising a hammock-like shallow-U looping from beneath the Y or the W respectively to the following vowel, together with a small Y or W letter respectively marked in, near or through the U;

wherein consonants before a vowel are ruled as to be spoken but are not marked as playable.

4. A method according to claim 2 comprising marking up the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

a direct link, comprising a liaison hammock strung beneath and between the linked letters to indicate that the spoken momentum should be carried from one linked letter to the next without interruption or a pause or break between words;

a play-and-link comprising a liaison hammock combined with single or double underlining of the first consonant to indicate that the first consonant is played and the second consonant is not; and

a prepare-and-link comprising a forward slash through a first consonant being a consonant to be prepared, combined with a liaison hammock to a second consonant to show the linkage between the first and second consonants, playable consonants being indicated with underlining;

wherein the method is implemented in a language selected from the group consisting of English;

American English;

French;

Spanish;

German;

Japanese;

Russian;

Chinese;

Arabic;

Hindi;

a written and spoken language having a graphic symbol set and a rules-based grammar; and

a dialect or a specialist subset of any one of the foregoing languages.

5. A method according to claim 4 employing an acoustic library comprising digitally recorded speech elements, being speech elements having been spoken with the prosody indicated by the graphic symbol markings, the method comprising utilizing the prosodic graphical symbols to promote generation of a humanized sound in synthesized speech output, the speech output optionally being formant speech, and comprising one or more, trained persons accurately pronouncing text according to the mark-up of the text with the prosodic graphical symbols to produce databases of pronounced speech containing pronounced speech accurately following the speech-code-marked text, digitally rendering the prosodic graphical symbols and employing the digitally rendered graphical symbols in synthesizer software for electronic markup of text to be machine spoken to facilitate the introduction into the digital domain of prosodic elements intended to be present the output speech.

6. A method according to claim 1 comprising marking the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

graphical symbols indicating pitch control required for pronunciation of a letter, diphthong, syllable or other speech element according to a desired prosody;

an upglide to indicate rising pitch;

a downglide to indicate descending pitch;

a circumflex to indicate a rising then descending pitch or a descending then rising pitch;

a level sustention to indicate unchanged pitch;

a forward slash through a first letter followed by a closely related or identical sounding consonant to indicate that the first letter is prepared;

a shallow U-shaped liaison hammock to indicate that letters separated from each other by other letters are to be pronounced in a continuous manner, without pause between them;

a single underline to mark as playable percussives including tympani drumbeats, D, B and G and snare, bass and tom-tom drumbeats, T, P and K, respectively, unmarked consonants being not playable;

a double underline to mark as playable strings N, M, V and Z woodwinds, L, NG, TH and ZH and sound effects F, S, SH and th, unmarked consonants being not playable;

the letter combination “

hw”

marked over or adjacent to the letter combination WH in the text to be spoken to indicate that the H should be sounded first, followed by the W, and that neither should be played; and

Y and W connectives to indicate that speech continuity should be maintained from one word to the next or from one syllable to the next when a [Y] or a [W] occurs before another vowel, the Y and W connectives each comprising a hammock-like shallow-U looping from beneath the Y or the W respectively to the following vowel, together with a small Y or W letter respectively marked in, near or through the U;

wherein consonants before a vowel are ruled as to be spoken but are not marked as playable.

7. A method according to claim 1 comprising marking up the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

a direct link, comprising a liaison hammock strung beneath and between the linked letters to indicate that the spoken momentum should be carried from one linked letter to the next without interruption or a pause or break between words;

a play-and-link comprising a liaison hammock combined with single or double underlining of the first consonant to indicate that the first consonant is played and the second consonant is not; and

a prepare-and-link comprising a forward slash through a first consonant being a consonant to be prepared, combined with a liaison hammock to a second consonant to show the linkage between the first and second consonants, playable consonants being indicated with underlining.

8. A method according to claim 1 comprising placing the prosodic graphical symbols adjacent to the text to be spoken, coordinating the prosodic graphical symbols in a row directly above the text, placing the prosodic graphical symbols selectively beneath the text or placing the prosodic graphical symbols both above the text and beneath the text.

9. A method according to claim 1 comprising marking up the text to be spoken by rendering the text in lines, marking the intelligibility pronunciation notations above the text, marking the prosodic graphical symbols beneath the text, marking a pitch reference line above the intelligibility pronunciation notation line and marking further prosodic symbols above the pitch reference line to indicate desired pitch changes and emphasis.

10. A method according to claim 9 wherein the pitch reference line comprises a Y-buzz pitch line and the method further comprises marking a desired intonation pattern above the Y-buzz pitch line using smaller dots to indicate less stress and larger dots to indicate more stress, the dots being positioned at levels above the Y-buzz pitch line to be indicative of the desired pitch relative to the speaker'"'"'s Y-buzz pitch line.

11. A method according to claim 1, for automated application of prosodic markings to text, the method comprising employing at least one computational linguistics algorithm to identify and mark the text to be pronounced to indicate desired prosodic pronunciation.

12. A method according to claim 11 comprising employing code variables corresponding with desired pronunciation sounds to generate values for acoustic variables employable to specify inputs to a speech synthesizer for outputting the marked-up text as synthesized speech.

13. A method according to claim 1 employing an acoustic library comprising digitally recorded speech elements, being speech elements having been spoken with the prosody indicated by the graphic symbol markings.

14. A method according to claim 13 implemented in a language selected from the group consisting of English;

American English;

French;

Spanish;

German;

Japanese;

Russian;

Chinese;

Arabic;

Hindi;

a written and spoken language having a graphic symbol set and a rules-based grammar; and

a dialect or a specialist subset of any one of the foregoing languages.

15. A method according to claim 1 comprising utilizing the prosodic graphical symbols to promote generation of a humanized sound in synthesized speech output, and comprising one or more, trained persons accurately pronouncing text according to the mark-up of the text with the prosodic graphical symbols to produce databases of pronounced speech containing pronounced speech accurately following the speech-code-marked text.

16. A method according to claim 15 comprising digitally rendering the prosodic graphical symbols and employing the digitally rendered graphical symbols in synthesizer software for electronic markup of text to be machine spoken to facilitate the introduction into the digital domain of prosodic elements intended to be present the output speech.

17. A method according to claim 16 wherein recorded speech corresponding to the graphical notations for text comprising one or more of letters, words, phrases, sentences, paragraphs and longer texts is digitized into databases and analyzed to provide algorithms or metrics to specify the relationship of specific phonetic data corresponding to specific text and the graphical notations related to the text.

18. A method according to claim 17 comprising utilizing one or more of the provided algorithms or metrics to provide input parameters to a speech synthesizer to re-create sounds that imitate human speech for a particular text to be synthesized as speech with a specified prosody.

19. A method according to claim 1 wherein the pitch change notations include a pitch reference, relative pitch notations and stress notations for the relative pitch notations.

20. A computerized speech synthesizer for synthesizing speech from text comprising a data processing unit, data memory, and a text input for inputting text to be synthesized to the speech synthesizer, and a speech synthesis program for:

electronically marking text input to the speech synthesizer with electronic versions of one or more prosodic graphical symbols, the electronically marked text being displayable or printable as human-readable text marked with the prosodic graphical symbols, to indicate to a speaker a desired speech characteristics to be employed in speaking the text, wherein the prosodic graphical symbols indicate a desired prosody and include intelligibility pronunciation notations in sequence with the text and pitch change notations in sequence with the text; and

generating a synthesized speech output comprising phonetic data corresponding with the marked up text and having the prosody indicated by the prosodic graphical symbols.