System and method for synthesizing human speech using multiple speakers and context

• US 9,368,104 B2
• Filed: 03/15/2013
• Issued: 06/14/2016
• Est. Priority Date: 04/30/2012
• Status: Active Grant

• Current Assignee: SRC, Inc.
• Original Assignee: SRC, Inc.
• Inventors: Eller, David Donald, Morphet, Steven Brian, Boyett, Watson Brent
• Primary Examiner(s): Opsasnick, Michael N
• Application Number:
  US13/835,315
• Publication Number:
  US 20130289998A1
• Time in Patent Office: 1,187 Days
• Field of Search: 704/258, 704/260, 704/266
• US Class Current: 1/1
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination

Abstract

A system and method for realistic speech synthesis which converts text into synthetic human speech with qualities appropriate to the context such as the language and dialect of the speaker, as well as expanding a speaker'"'"'s phonetic inventory to produce more natural sounding speech.

Claims

1. A method of synthesizing speech from text, comprising the steps of:

receiving text from which speech will be synthesized;

selecting, based on the received text, one or more scenario parameters, wherein the one or more scenario parameters are selected from the group consisting of language, dialect, accent, phonetic reduction, domain, context, and speaker number;

identifying text metadata within the received text, wherein the text metadata comprises elements other than words within the text;

parsing the received text, other than the identified text metadata, into a plurality of corresponding phonetic components;

merging said plurality of phonetic components with breathing and non-speech effects to produce a transcript of phoneme segment strings corresponding to the received text;

producing prosody contour data from said one or more selected scenario parameters and said transcript of phoneme segment strings;

producing stitched filter data from said one or more selected scenario parameters and said transcript of phoneme segment strings;

synthesizing speech from said stitched filter data and said prosody contour data; and

outputting said synthesized speech from a playback device.

2. The method according to claim 1, wherein the one or more scenario parameters are selected by a user.

3. The method according to claim 1, wherein a user provides the prosody contour.

4. The method according to claim 1, wherein producing said stitched filter data comprises:

receiving said text parsed into corresponding phonetic components;

matching each corresponding phonetic component with a corresponding signal feature candidate;

identifying within the corresponding signal feature candidates each pair of adjacent signal feature candidates;

modifying the formant features of each phonetic component within the pair of adjacent signal feature candidates such the first candidate within the pair transitions smoothly to the second phonetic candidate within the pair.

5. A method for synthesizing speech from text, comprising the steps of:

providing a computer having a first database and a second database stored in the memory thereof and in which data is stored, said data in said first database representing a set of signal feature candidates representative of a single speaker, and said data in said second database representing a second set of signal feature candidates representative of multiple speakers;

receiving text from which speech will be synthesized;

selecting, based on the received text, one or more scenario parameters, wherein the one or more scenario parameters are selected from the group consisting of language, dialect, accent, phonetic reduction, domain, context, and speaker number;

identifying text metadata within the received text, wherein the text metadata comprises elements other than words within the text;

parsing the received text, other than the identified text metadata, into a plurality of target phonetic components;

analyzing said single speaker signal feature candidates from said first database to determine whether a corresponding single speaker signal feature candidate exists for each target phonetic component;

retrieving from said second database a replacement signal feature candidate from said second set of signal feature candidates for any target phonetic component that does not have a corresponding single speaker signal feature candidate;

synthesizing speech from at least one of said corresponding single signal feature candidates and said replacement signal feature candidates, the speech comprising prosody contour data and stitched filter data from said one or more selected scenario parameters.

6. The method according to claim 5, further comprising the step of:

modifying said replacement signal feature candidates such that the synthesized speech from the replacement signal feature candidates resembles the synthesized speech of the single speaker signal feature candidates.

7. The method according to claim 6, wherein modifying comprises:

constructing a map of said single speaker signal feature candidates and corresponding signal feature candidates from said second set of signal feature candidates;

training a system on said map capable of generalizing difference between said single speaker signal feature candidates and corresponding signal feature candidates from said second stored set of signal feature candidates;

modifying said replacement phonetic component according to generalized difference represented in said system.

8. The method according to claim 5, wherein said single speaker signal feature candidates and said replacement components are signal feature candidates representing diphones, each candidate from said single speaker signal feature candidates and said replacement components having a first steady-state portion, a transition portion, and a second steady-state portion.

9. The method according to claim 8, wherein modifying comprises:

identifying said transition portion of said replacement diphones and said single speaker candidate;

training a system on said transition portion, capable of generalizing features of said transition portion;

generating a new transition portion according said system;

replacing said transition portion of said first steady-portion with said new transition portion.

10. A method for synthesizing speech from text, comprising the steps of:

providing a computer having a first database and a second database stored in the memory thereof and in which data is stored, said data in said first database representing a set of signal feature candidates representative of a single speaker, and said data in said second database representing a second set of signal feature candidates;

receiving text from which speech will be synthesized;

selecting, based on the received text, one or more scenario parameters, wherein the one or more scenario parameters are selected from the group consisting of language, dialect, accent, phonetic reduction, domain, context, and speaker number;

identifying text metadata within the received text, wherein the text metadata comprises elements other than words within the text;

parsing the received text, other than the identified text metadata, into a plurality of target phonetic components;

analyzing said single speaker signal feature candidates from said first database to determine whether a corresponding single speaker signal feature candidate of sufficient quality exists for each target phonetic component;

retrieving from said second database a replacement signal feature candidate from said second set of signal feature candidates for any target phonetic component that does not have a corresponding single speaker signal feature candidate of sufficient quality; and

synthesizing speech from at least one of the corresponding single speaker signal feature candidates and the replacement signal feature candidates, the speech comprising prosody contour data and stitched filter data from said one or more selected scenario parameters.

11. The method according to claim 10, wherein sufficient quality is determined by:

receiving said single speaker signal feature candidates;

identifying within the corresponding signal feature candidates each pair of adjacent signal feature candidates;

measuring a cost of joining each said pair of adjacent signal feature candidates; and

determining whether said cost is too high.

12. The method according to claim 10, further comprising the step of:

modifying said replacement signal feature candidates such that the resulting synthesized speech from the replacement signal feature candidates resembles the resulting synthesized speech of the single speaker signal feature candidates.

13. The method according to claim 12, wherein modifying further comprises:

constructing a map of said single speaker signal feature candidates and corresponding signal feature candidates from said second set of signal feature candidates;

training a system on said map capable of generalizing difference between said single speaker signal feature candidates and corresponding signal feature candidates from said second s set of signal feature candidates;

modifying said replacement signal feature candidate according to generalized difference represented in said system.

14. The method according to claim 12, wherein said single speaker signal feature candidates and said replacement components are signal feature candidates representing diphones;

each candidate from said single speaker phonetic components and said replacement components having a first steady-state portion, a transition portion, and a second steady-state portion.

15. The method according to claim 14, wherein modifying further comprises:

identifying said transition portion of said replacement diphones and said single speaker feature candidates;

training a system on said transition portion, capable of generalizing features of said transition portion;

generating a new transition portion according said system;

replacing said transition portion of said first steady-portion with said new transition portion.

16. A non-transitory computer-readable storage medium containing program code comprising:

program code for receiving text from which speech with be synthesized;

program code for selecting, based on the received text, one or more scenario parameters;

program code for identifying text metadata within the received text, wherein the text metadata comprises elements other than words within the text;

program code for parsing the received text, other than the identified text metadata, into a plurality of corresponding phonetic components;

program code for merging said plurality of phonetic components with breathing and non-speech effects to produce a transcript of phoneme segment strings corresponding to the received text;

program code for producing prosody contour data from said one or more selected scenario parameters and said transcript of phoneme segment strings;

program code for producing stitched filter data from said one or more selected scenario parameters and said transcript of phoneme segment strings;

program code for synthesizing speech from said stitched filter data and said prosody contour data;

program code for outputting said synthesized speech from a playback device.

17. The storage medium according to claim 16, wherein the one or more scenario parameters are selected from the group consisting of language, dialect, accent, phonetic reduction, domain, context, and a single speaker.

18. The storage medium according to claim 16, wherein the one or more scenario parameters are selected by a user.

19. The storage medium according to claim 16, wherein the user provides the prosody contour.

20. The storage medium according to claim 16, wherein producing said stitched filter data further comprises:

program code for receiving said text parsed into corresponding phonetic components;

program code for matching each phonetic component with a corresponding signal feature candidate;

program code for identifying within the corresponding signal feature candidates each pair of adjacent signal feature candidates;

program code for modifying the formant features of each signal feature candidate within the pair of adjacent signal feature candidates such the first phonetic component within the pair transitions smoothly to the second phonetic component within the pair.

21. A non-transitory computer-readable storage medium containing program code, comprising:

program code for receiving text from which speech will be synthesized;

program code for selecting, based on the received text, one or more scenario parameters, wherein the one or more scenario parameters are selected from the group consisting of language, dialect, accent, phonetic reduction, domain, context, and speaker number;

program code for identifying text metadata within the received text, wherein the text metadata comprises elements other than words within the text;

program code for parsing the received text, other than the identified text metadata, into a plurality of target phonetic components;

program code for analyzing a single speaker'"'"'s signal feature candidates, said single speaker'"'"'s signal feature candidates stored in a database, to determine whether a corresponding single speaker signal feature candidate exists for each said target phonetic component;

program code for retrieving from a second set of signal feature candidates representative of multiple speakers'"'"' signal feature candidates, said second set of signal feature candidates stored in database, a replacement signal feature candidate for any target phonetic component that does not have a corresponding single speaker signal feature candidate;

program code for synthesizing speech from at least one of said corresponding single speaker signal feature candidates and said replacement signal feature candidates, the speech comprising prosody contour data and stitched filter data from said one or more selected scenario parameters.

22. The storage medium according to claim 21, further comprising program code for modifying said replacement signal feature candidates such that the synthesized speech from the replacement signal feature candidates resembles the synthesized speech of the single speaker signal feature candidates.

23. The storage medium according to claim 22, wherein said program code for modifying comprises:

program code for constructing a map of said single speaker signal feature candidates and corresponding signal feature candidates from said second set of signal feature candidates;

program code for training a system on said map capable of generalizing difference between said single speaker signal feature candidates and corresponding signal feature candidates from said second stored set of signal feature candidates;

program code for modifying said replacement signal feature candidate according to generalized difference represented in said system.

24. The storage medium according to claim 21, wherein said single speaker signal feature candidates and said replacement components are signal feature candidate representing diphones, each candidate from said single speaker signal feature candidates and said replacement components having a first steady-state portion, a transition portion, and a second steady-state portion.

25. The storage medium according to claim 24, wherein program code for modifying comprises:

program code for identifying said transition portion of said replacement candidates and said single speaker candidate;

program code for training a system on said transition portion, capable of generalizing features of said transition portion;

program code for generating a new transition portion according said system;

program code for replacing said transition portion of said first steady-portion with said new transition portion.

26. A non-transitory computer-readable storage medium containing program code, comprising:

program code for receiving text from which speech will be synthesized;

program code for selecting, based on the received text, one or more scenario parameters, wherein the one or more scenario parameters are selected from the group consisting of language, dialect, accent, phonetic reduction, domain, context, and speaker number;

program code for identifying text metadata within the received text, wherein the text metadata comprises elements other than words within the text;

program code for parsing the received text, other than the identified text metadata, into a plurality of target phonetic components;

program code for analyzing a single speaker'"'"'s signal feature candidates, said single speaker'"'"'s signal feature candidates stored in a database, to determine whether a corresponding single speaker signal feature candidate of sufficient quality exists for each said target phonetic component;

program code for retrieving from a second set of signal feature candidates, said second set of signal feature candidates stored in database, a replacement signal feature candidate for any target phonetic component that does not have a corresponding single speaker signal feature candidate of sufficient quality;

program code for synthesizing speech from at least one of said corresponding single speaker signal feature candidates and said replacement signal feature candidates, the speech comprising prosody contour data and stitched filter data from said one or more selected scenario parameters.

27. The storage medium according to claim 26, wherein sufficient quality is determined by program code comprising:

program code for receiving said single speaker signal feature candidates;

program code for identifying within the corresponding signal feature candidates each pair of adjacent signal feature candidates;

program code for measuring a cost of joining each said pair of adjacent signal feature candidates; and

program code for determining whether said cost is too high.

28. The storage medium according to claim 26, further comprising program code for modifying said replacement signal feature candidates such that the synthesized speech from the replacement signal feature candidates resembles the synthesized speech of the single speaker signal feature candidates.

29. The storage medium according to claim 28, wherein program code for modifying further comprises:

program code for constructing a map of said single speaker signal feature candidates and corresponding signal feature candidates from said second set of signal feature candidates;

program code for training a system on said map capable of generalizing difference between said single speaker signal feature candidates and corresponding signal feature candidates from said second s set of signal feature candidates;

program code for modifying said replacement signal feature candidate according to generalized difference represented in said system.

30. The storage medium according to claim 28, wherein said single speaker signal feature candidates and said replacement components are diphones;

each diphone from said single speaker signal feature candidates and said replacement components having a first steady-state portion, a transition portion, and a second steady-state portion.

31. The storage medium according to claim 30, wherein program code for modifying further comprises:

program code for identifying said transition portion of said replacement diphones and said single speaker feature candidates;

program code for training a system on said transition portion, capable of generalizing features of said transition portion;

program code for generating a new transition portion according said system;

program code for replacing said transition portion of said first steady-portion with said new transition portion.