Text to speech synthesis for texts with foreign language inclusions

• US 8,990,089 B2
• Filed: 11/19/2012
• Issued: 03/24/2015
• Est. Priority Date: 11/17/2011
• Status: Active Grant

• Current Assignee: Cerence Operating Company (Cerence Inc.)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Wouters, Johan, Traber, Christof, Wilpert, Alexis, Hagstrand, David, Keller, Jrgen, Nozhov, Igor
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US13/680,260
• Publication Number:
  US 20130132069A1
• Time in Patent Office: 855 Days
• Field of Search: 704/8, 704/258, 704/260, 704/269
• US Class Current: 704/260
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 13/06 Elementary speech units used in speech synthesisers; Concatenation rules
• CPC Class Codes: G10L 13/086 Detection of language

Abstract

A speech output is generated from a text input written in a first language and containing inclusions in a second language. Words in the native language are pronounced with a native pronunciation and words in the foreign language are pronounced with a proficient foreign pronunciation. Language dependent phoneme symbols generated for words of the second language are replaced with language dependent phoneme symbols of the first language, where said replacing includes the steps of assigning to each language dependent phoneme symbol of the second language a language independent target phoneme symbol, mapping to each one language independent target phoneme symbol a language independent substitute phoneme symbol assignable to a language dependent substitute phoneme symbol of the first language, substituting the language dependent phoneme symbols of the second language by the language dependent substitute phoneme symbols of the first language.

Claims

1. A computer-implemented method employing at least one hardware implemented computer processor for converting an input text with words containing at least one word in a first language and at least one word in at least one further language into a speech waveform comprising the steps of:

providing a phoneme symbol resource of the first language;

generating a sequence of language dependent phoneme symbols including the steps of;

identifying at least one specific further language for words which do not belong to the first language, adding at least one phoneme symbol resource of the at least one specific further language, applying the phoneme symbol resource of the first language for words in the first language, and applying the at least one phoneme symbol resource of the at least one specific further language for words in the at least one specific further language;

replacing language dependent phoneme symbols generated for the at least one word of the at least one further language with language dependent phoneme symbols of the first language, including the steps of;

assigning to each language dependent phoneme symbol of the at least one specific further language a language independent target phoneme symbol, mapping to each language independent target phoneme symbol a language independent substitute phoneme symbol assignable to a language dependent substitute phoneme symbol of the first language, and substituting the language dependent phoneme symbols of the at least one specific further language by the language dependent substitute phoneme symbols of the first language which are assignable to the language independent substitute phoneme symbols;

deriving at least one target unit sequence corresponding to a sequence of phoneme symbols including language dependent phoneme symbols of the first language;

selecting from a waveform unit database of the first language a waveform unit sequence approximating the at least one target unit sequence; and

concatenating the waveform unit sequence to the speech waveform.

2. The method according to claim 1, wherein the step of mapping includes the steps of:

searching for the language independent target phoneme symbol in a predefined table of distances between language independent phoneme symbols at least one language independent approximate phoneme symbol with a distance below a given maximum distance, checking whether the at least one language independent approximate phoneme symbol can be assigned to a language dependent phoneme symbol of the first language, and selecting from the at least one assignable language independent approximate phoneme symbol at least one language independent substitute phoneme symbol.

3. The method according to claim 1, wherein the input text is enriched with language tags.

4. The method according to claim 1, wherein language tags are added to the input text by a detection algorithm that splits and assigns the input text into text parts belonging to the first language and text parts belonging to the at least one further foreign language.

5. The method according to claim 4, wherein the steps of splitting and assigning include at least one of the steps of:

identifying a change of character sets in the input text, using the probability of character n-grams, applying predetermined language transition probabilities, and taking into account punctuations.

6. The method according to claim 1, wherein the steps of applying the phoneme symbol resource of the first language for words in the first language and applying the at least one phoneme symbol resource of the at least one specific further language for words in the at least one specific further language are applied in series.

7. The method according to claim 6, wherein within a sentence of the input text the step of applying the phoneme symbol resource of the first language is made after the step of applying the at least one phoneme symbol resource of the at least one specific further language for words in the at least one specific further language.

8. The method according to claim 6, wherein at the same time only one phoneme symbol resource is active.

9. The method according to claim 1, wherein the language independent target phoneme symbols belong to a set of X-SAMPA phonetic transcriptions.

10. The method according to claim 9, wherein for the step of mapping to a language independent target phoneme symbol a language independent substitute phoneme symbol assignable to a language dependent substitute phoneme symbol of the first language predefined distances between pairs of X-SAMP consonants and between pairs of X-SAMP vowels are used to find a language independent substitute phoneme symbol with the shortest distance to the language independent target phoneme symbol.

11. A computer program product encoded in a non-transitory computer-readable medium for converting an input text with words containing at least one word in a first language and at least one word in at least one further language into a speech waveform, the product comprising:

program code for providing a phoneme symbol resource of the first language;

program code for generating a sequence of language dependent phoneme symbols including program code for;

identifying at least one specific further language for words which do not belong to the first language, adding at least one phoneme symbol resource of the at least one specific further language, applying the phoneme symbol resource of the first language for words in the first language, and applying the at least one phoneme symbol resource of the at least one specific further language for words in the at least one specific further language;

program code for replacing language dependent phoneme symbols generated for the at least one word of the at least one further language with language dependent phoneme symbols of the first language, including program code for;

assigning to each language dependent phoneme symbol of the at least one specific further language a language independent target phoneme symbol, mapping to each language independent target phoneme symbol a language independent substitute phoneme symbol assignable to a language dependent substitute phoneme symbol of the first language, and substituting the language dependent phoneme symbols of the at least one specific further language by the language dependent substitute phoneme symbols of the first language which are assignable to the language independent substitute phoneme symbols;

program code for deriving at least one target unit sequence corresponding to a sequence of phoneme symbols including language dependent phoneme symbols of the first language;

program code for selecting from a waveform unit database of the first language a waveform unit sequence approximating the at least one target unit sequence; and

program code for concatenating the waveform unit sequence to the speech waveform.

12. A text processor for converting an input text with words containing at least one word in a first language and at least one word in at least one further language into a speech waveform, said processor comprising means for:

providing a phoneme symbol resource of the first language;

generating a sequence of language dependent phoneme symbols including means for;

identifying at least one specific further language for words which do not belong to the first language, adding at least one phoneme symbol resource of the at least one specific further language, applying the phoneme symbol resource of the first language for words in the first language, and applying the at least one phoneme symbol resource of the at least one specific further language for words in the at least one specific further language;

replacing language dependent phoneme symbols generated for the at least one word of the at least one further language with language dependent phoneme symbols of the first language, including means for;

assigning to each language dependent phoneme symbol of the at least one specific further language a language independent target phoneme symbol, mapping to each language independent target phoneme symbol a language independent substitute phoneme symbol assignable to a language dependent substitute phoneme symbol of the first language, and substituting the language dependent phoneme symbols of the at least one specific further language by the language dependent substitute phoneme symbols of the first language which are assignable to the language independent substitute phoneme symbols;

deriving at least one target unit sequence corresponding to a sequence of phoneme symbols including language dependent phoneme symbols of the first language;

selecting from a waveform unit database of the first language a waveform unit sequence approximating the at least one target unit sequence; and

concatenating the waveform unit sequence to the speech waveform.