Prosodic Speech Text Codes and Their Use in Computerized Speech Systems

• US 20070260461A1
• Filed: 03/07/2005
• Published: 11/08/2007
• Est. Priority Date: 03/05/2004
• Status: Active Grant

• Current Assignee: Lessac Technologies, Inc.
• Original Assignee: Lessac Technologies, Inc.
• Inventors: Kur, Barry, Krebs, Nancy, Gaary, Diane, Marple, Gary, Wilson, H. Donald, Park, Sue, Wilson, Mary Louise
• Granted Patent: US 7,877,259 B2
• Time in Patent Office: Days
• US Class Current: 704/260
• CPC Class Codes: G09B 5/04 with audible presentation of the material to be studied sound recording or reproducing G11B
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser
• CPC Class Codes: G10L 13/10 Prosody rules derived from text; Stress or intonation

Abstract

Disclosed are a method of, and system for, acoustically coding text for use in the synthesis of speech from the text, the method comprising marking the text to be spoken with one or more graphic symbols to indicate to a speaker a desired prosody to impart to the spoken text to convey expressive meaning. The markups can comprise grapheme-phoneme pairs each comprising a visible prosodic-indicating grapheme employable with written text and a corresponding digital phoneme functional in the digital domain. The invention is useful in the generation of appealing, humanized machine speech for a wide range of applications, including voice mail systems, electronically enabled appliances, automobiles, computers, robotic assistants, games and the like, in spoken books and magazines, drama and other entertainment.

Claims

1-17. -17. (canceled)

18. A method of marking up text for use in the synthesis of speech from the text, the method comprising marking the text to be spoken with one or more graphic symbols to indicate to a speaker desired speech characteristics to be employed in speaking the text wherein the graphic symbols comprise acoustic codes indicating a desired prosody to be imparted to the spoken text by the speaker.

19. A method according to claim 18 wherein the imparted prosody comprises one or more prosody elements selected from the group consisting of pace, intonation pattern, rhythm, musicality, amplitude, pauses for emphasis and breath, and formal and informal articulations of words and phrases.

20. A method according to claim 19 comprising marking visible text with graphic prosody symbols or electronically marking electronic text with electronic versions of the graphic symbols, the electronically marked text being displayable or printable as human-readable graphically marked text and being effective for communicating desired prosody to a speech synthesizer in a manner permitting control of the prosody of the output speech.

21. A method according to claim 18 comprising marking the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

graphical symbols indicating pitch control required for pronunciation of a letter, diphthong, syllable or other speech element according to a desired prosody;

an upglide to indicate rising pitch;

a downglide to indicate descending pitch;

a circumflex to indicate a rising then descending pitch or a descending then rising pitch;

a level sustention to indicate unchanged pitch;

a forward slash through a first letter followed by a closely related or identical sounding consonant to indicate that the first letter is prepared;

a shallow U-shaped liaison hammock to indicate that letters separated from each other by other letters are to be pronounced in a continuous manner, without pause between them;

a single underline to mark as playable percussives including tympani drumbeats, D, B and G and snare, bass and tom-tom drumbeats, T, P and K, respectively, unmarked consonants being not playable;

a double underline to mark as playable strings N, M, V and Z woodwinds, L, NG, TH and ZH and sound effects F, S, SH and th, unmarked consonants being not playable;

the letter combination “

hw”

marked over or adjacent to the letter combination WH in the text to be spoken to indicate that the H should be sounded first, followed by the W, and that neither should be played; and

Y and W connectives to indicate that speech continuity should be maintained from one word to the next or from one syllable to the next when a [Y] or a [w] occurs before another vowel, the Y and W connectives each comprising a hammock-like shallow-U looping from beneath the Y or the W respectively to the following vowel, together with a small Y or W letter respectively marked in, near or through the U;

wherein consonants before a vowel are ruled as to be spoken but are not marked as playable.

22. A method according to claim 18 comprising marking up the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

a direct link, comprising a liaison hammock strung beneath and between the linked letters to indicate that the spoken momentum should be carried from one linked letter to the next without interruption or a pause or break between words;

a play-and-link comprising a liaison hammock combined with single or double underlining of the first consonant to indicate that the first consonant is played and the second consonant is not; and

a prepare-and-link comprising a forward slash through a first consonant being a consonant to be prepared, combined with a liaison hammock to a second consonant to show the linkage between the first and second consonants, playable consonants being indicated with underlining.

23. A method according to claim 18 comprising placing the prosodic graphical symbols adjacent to the text to be spoken by coordinating the prosodic graphical symbols in a row directly above the text, by placing the prosodic graphical symbols selectively beneath the text or by placing the prosodic graphical symbols both above the text and beneath the text.

24. A method according to claim 18 comprising marking up the text to be spoken by rendering the text in lines, marking intelligibility pronunciation notations above the text, marking prosodic graphical symbols beneath the text, marking a pitch reference line above the intelligibility pronunciation notation line and marking further prosodic symbols above the pitch reference line to indicate desired pitch changes and emphasis.

25. A method according to claim 24 wherein the pitch reference line comprises a Y-buzz pitch line and the method further comprises marking a desired intonation pattern above the Y-buzz pitch line using smaller dots to indicate less stress and larger dots to indicate more stress, the dots being positioned at levels above the Y-buzz pitch line to be indicative of the desired pitch relative to the speaker'"'"'s Y-buzz pitch line.

26. A method according to claim 18, for automated application of prosodic markings to text, the method comprising employing at least one computational linguistics algorithm to identify and mark the text to be pronounced to indicate desired prosodic pronunciation.

27. A method according to claim 26 comprising employing code variables corresponding with desired pronunciation sounds to generate values for acoustic variables employable to specify inputs to a speech synthesizer for outputting the marked-up text as synthesized speech.

28. A method according to claim 18 employing an acoustic library comprising digitally recorded speech elements, being speech elements having been spoken with the prosody indicated by the graphic symbol markings.

29. A method according to claim 28 implemented in a language selected from the group consisting of English;

American English;

French;

Spanish;

German;

Japanese;

Russian;

Chinese;

Arabic;

Hindi;

a written and spoken language having a graphic symbol set and a rules-based grammar; and

a dialect or a specialist subset of any one of the foregoing languages.

30. A method according to claim 18 comprising utilizing the prosodic graphical symbols to promote generation of a humanized sound in synthesized speech output, the speech output optionally being formant speech, and comprising one or more, trained persons accurately pronouncing text according to the mark-up of the text with the prosodic graphical symbols to produce databases of pronounced speech containing pronounced speech accurately following the speech-code-marked text.

31. A method according to claim 30 comprising digitally rendering the prosodic graphical symbols and employing the digitally rendered graphical symbols in synthesizer software for electronic markup of text to be machine spoken to facilitate the introduction into the digital domain of prosodic elements intended to be present the output speech.

32. A method according to claim 31 wherein recorded speech corresponding to the graphical notations for text comprising one or more of letters, words, phrases, sentences, paragraphs and longer texts is digitized into databases and analyzed to provide algorithms or metrics to specify the relationship of specific phonetic data corresponding to specific text and the graphical notations related to the text.

33. A method according to claim 32 comprising utilizing one or more of the provided algorithms or metrics to provide input parameters to a speech synthesizer to re-create sounds that imitate human speech for a particular text to be synthesized as speech with a specified prosody.

34. A method according to claim 19 comprising marking the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

graphical symbols indicating pitch control required for pronunciation of a letter, diphthong, syllable or other speech element according to a desired prosody;

an upglide to indicate rising pitch;

a downglide to indicate descending pitch;

a circumflex to indicate a rising then descending pitch or a descending then rising pitch;

a level sustention to indicate unchanged pitch;

a forward slash through a first letter followed by a closely related or identical sounding consonant to indicate that the first letter is prepared;

a shallow U-shaped liaison hammock to indicate that letters separated from each other by other letters are to be pronounced in a continuous manner, without pause between them;

a single underline to mark as playable percussives including tympani drumbeats, D, B and G and snare, bass and tom-tom drumbeats, T, P and K, respectively, unmarked consonants being not playable;

a double underline to mark as playable strings N, M, V and Z woodwinds, L, NG, TH and ZH and sound effects F, S, SH and th, unmarked consonants being not playable;

the letter combination “

hw”

marked over or adjacent to the letter combination WH in the text to be spoken to indicate that the H should be sounded first, followed by the W, and that neither should be played; and

Y and W connectives to indicate that speech continuity should be maintained from one word to the next or from one syllable to the next when a [Y] or a [W] occurs before another vowel, the Y and W connectives each comprising a hammock-like shallow-U looping from beneath the Y or the W respectively to the following vowel, together with a small Y or W letter respectively marked in, near or through the U;

wherein consonants before a vowel are ruled as to be spoken but are not marked as playable.

35. A method according to claim 19 comprising marking up the text to be spoken with one, more than one or all of the prosodic graphical symbols selected from the group consisting of:

a direct link, comprising a liaison hammock strung beneath and between the linked letters to indicate that the spoken momentum should be carried from one linked letter to the next without interruption or a pause or break between words;

a play-and-link comprising a liaison hammock combined with single or double underlining of the first consonant to indicate that the first consonant is played and the second consonant is not; and

a prepare-and-link comprising a forward slash through a first consonant being a consonant to be prepared, combined with a liaison hammock to a second consonant to show the linkage between the first and second consonants, playable consonants being indicated with underlining;

wherein the method is implemented in a language selected from the group consisting of English;

American English;

French;

Spanish;

German;

Japanese;

Russian;

Chinese;

Arabic;

Hindi;

a written and spoken language having a graphic symbol set and a rules-based grammar; and

a dialect or a specialist subset of any one of the foregoing languages.

36. A method according to claim 35 employing an acoustic library comprising digitally recorded speech elements, being speech elements having been spoken with the prosody indicated by the graphic symbol markings, the method comprising utilizing the prosodic graphical symbols to promote generation of a humanized sound in synthesized speech output, the speech output optionally being formant speech, and comprising one or more, trained persons accurately pronouncing text according to the mark-up of the text with the prosodic graphical symbols to produce databases of pronounced speech containing pronounced speech accurately following the speech-code-marked text, digitally rendering the prosodic graphical symbols and employing the digitally rendered graphical symbols in synthesizer software for electronic markup of text to be machine spoken to facilitate the introduction into the digital domain of prosodic elements intended to be present the output speech.

37. A speech synthesizer controlled by acoustic coding variables input to the speech synthesizer wherein the acoustic coding variables correspond to prosodic specifications employed to generate recorded human speech having a desired prosodic pronunciation and wherein the recorded human speech is employed to provide a synthesized speech output embodying the desired prosodic pronunciation.