Methods and systems for automated generation of nativized multi-lingual lexicons

• US 8,768,704 B1
• Filed: 10/14/2013
• Issued: 07/01/2014
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Fructuoso, Javier Gonzalvo, Agiomyrgiannakis, Ioannis
• Primary Examiner(s): ROBERTS, SHAUN A
• Application Number:
  US14/053,052
• Time in Patent Office: 260 Days
• Field of Search: 704258-269, 704/277
• US Class Current: 704/260
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G10L 13/08 Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
• CPC Class Codes: G10L 13/086 Detection of language
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 2015/0633 using lexical or orthographic knowledge sources

Abstract

An input signal that includes linguistic content in a first language may be received by a computing device. The linguistic content may include text or speech. Based on an acoustic feature comparison between a plurality of first-language speech sounds and a plurality of second-language speech sounds, the computing device may associate the linguistic content in the first language with one or more phonemes from a second language. The computing device may also determine a phonemic representation of the linguistic content in the first language based on use of the one or more phonemes from the second language. The phonemic representation may be indicative of a pronunciation of the linguistic content in the first language according to speech sounds of the second language.

Claims

1. A method comprising:

receiving an input signal that includes linguistic content in a first language, wherein the linguistic content includes text or speech;

based on an acoustic feature comparison between a plurality of first-language speech sounds and a plurality of second-language speech sounds, associating, by a computing device that includes one or more processors, the linguistic content in the first language with one or more phonemes from a second language; and

determining, by the computing device, a phonemic representation of the linguistic content in the first language based on use of the one or more phonemes from the second language, wherein the phonemic representation is indicative of a pronunciation of the linguistic content in the first language according to speech sounds of the second language.

2. The method of claim 1, further comprising:

determining, by the computing device, cross-lingual assignments between the plurality of first-language speech sounds and a plurality of second-language phonemes, wherein associating the linguistic content in the first language with the one or more phonemes from the second language is based on the cross-lingual assignments.

3. The method of claim 2, further comprising:

determining, by the computing device, first-language assignments between the plurality of first-language speech sounds and a plurality of first-language phonemes, and second-language assignments between the plurality of second-language speech sounds and the plurality of second-language phonemes; and

determining a mapping configured to associate the plurality of first-language speech sounds with the plurality of second-language speech sounds, wherein determining the cross-lingual assignments is based on the mapping and the second-language assignments.

4. The method of claim 3, further comprising:

receiving configuration input comprising a first audio signal that includes speech content in the first language, a sequence of first-language phonemes that are assigned to a portion of the first audio signal, a second audio signal that includes speech content in the second language, and a sequence of second-language phonemes that are assigned to a portion of the second audio signal; and

based on the configuration input, determining the plurality of first-language speech sounds, the plurality of first-language phonemes, the plurality of second-language speech sounds, the plurality of second-language phonemes, the first-language assignments, and the second-language assignments.

5. The method of claim of claim 3, wherein determining the mapping comprises:

determining a first vector that includes representations of first acoustic features in the plurality of first-language speech sounds, and a second vector that includes representations of second acoustic features in the plurality of second-language speech sounds;

determining a third vector that includes association probabilities between the first vector and the second vector; and

determining the mapping based on the third vector.

6. The method of claim 5, wherein determining the mapping comprises a linear regression that includes repeating the determining of the third vector and the determining of the mapping based on the third vector until convergence of linear regression variables in the linear regression.

7. The method of claim 5, wherein the representations of the first-language speech sounds are indicative of Gaussian density functions (GDFs) that estimate one or more spectral features of the first-language speech sounds, wherein the representations of the second-language speech sounds are indicative of GDFs that estimate one or more spectral features of the second-language speech sounds, wherein determining the association probabilities in the third vector comprises identifying a given mean value of GDF in the first vector proximal to a given mean value of GDF in the second vector.

8. The method of claim 7, wherein determining the association probabilities in the third vector is further based on a phone mapping table or Viterbi decoding.

9. The method of claim 1, wherein the linguistic content corresponds to text in the first language, the method further comprising:

generating a synthetic audio signal based on the determined phonemic representation, wherein the synthetic audio signal is configured to correspond to the pronunciation of the text in the first language according to speech sounds of the second language.

10. The method of claim 1, wherein the linguistic content corresponds to speech in the first language that has the pronunciation according to speech sounds of the second language, the method further comprising:

generating text from the first language based on the determined phonemic representation, wherein the generated text is configured to correspond to the pronunciation of the speech in the first language according to speech sounds of the second language.

11. A method comprising:

receiving configuration input that includes first linguistic content in a first language and second linguistic content in a second language, wherein the first linguistic content includes a plurality of first-language speech sounds assigned to a plurality of first-language phonemes, and wherein the second linguistic content includes a plurality of second-language speech sounds assigned to a plurality of second-language phonemes;

based on an acoustic feature comparison between the plurality of first-language speech sounds and the plurality of second-language speech sounds, determining, by a computing device that includes one or more processors, assignments between the plurality of first-language speech sounds and the plurality of second-language phonemes; and

based on the assignments, generating a lexicon that includes a plurality of entries, wherein a given entry in the lexicon includes text in the first language matched with a phonemic representation that includes one or more of the plurality of second-language phonemes, wherein the phonemic representation is indicative of a pronunciation of the text in the first language according to speech sounds of the second language.

12. The method of claim 11, further comprising:

determining, by the computing device, a mapping configured to associate the plurality of first-language speech sounds with the plurality of second-language speech sounds, wherein determining the assignments is based on the mapping.

13. The method of claim of claim 12, wherein determining the mapping comprises:

determining a first vector that includes representations of first acoustic features in the plurality of first-language speech sounds, and a second vector that includes representations of second acoustic features in the plurality of second-language speech sounds;

determining a third vector that includes association probabilities between the first vector and the second vector; and

determining the mapping based on the third vector.

14. The method of claim 13, wherein determining the mapping comprises a linear regression that includes repeating the determining of the third vector and the determining of the mapping based on the third vector until convergence of linear regression variables in the linear regression.

15. The method of claim 11, further comprising:

receiving an input signal that includes the text in the first language; and

based on the phonemic representation in the lexicon that matches the text, generating a synthetic audio signal comprising a combination of one or more of the plurality of second-language speech sounds, wherein the synthetic audio signal is indicative of the pronunciation of the text in the first language according to speech sounds of the second language.

16. The method of claim 11, further comprising:

receiving an input signal that includes speech in the first language that has pronunciation according to speech sounds of the second language;

based on the assignments, determining one or more of the plurality of second-language phonemes that correspond to the speech in the input signal; and

based on an identification of the phonemic representation in the lexicon that includes the one or more determined second-language phonemes, providing the text in the first language that matches the phonemic representation, wherein the provided text from the lexicon corresponds to the speech in the input signal.

17. A non-transitory computer readable medium having stored therein instructions, that when executed by a computing device, cause the computing device to perform functions, the functions comprising:

receiving an input signal that includes linguistic content in a first language, wherein the linguistic content includes text or speech;

based on an acoustic feature comparison between a plurality of first-language speech sounds and a plurality of second-language speech sounds, associating the linguistic content in the first language with one or more phonemes from a second language; and

determining a phonemic representation of the linguistic content in the first language based on use of the one or more phonemes from the second language, wherein the phonemic representation is indicative of a pronunciation of the linguistic content in the first language according to speech sounds of the second language.

18. The non-transitory computer readable medium of claim 17, the functions further comprising:

determining cross-lingual assignments between the plurality of first-language speech sounds and a plurality of second-language phonemes, wherein associating the linguistic content in the first language with the one or more phonemes from the second language is based on the cross-lingual assignments.

19. The non-transitory computer readable medium of claim 18, the functions further comprising:

determining first-language assignments between the plurality of first-language speech sounds and a plurality of first-language phonemes, and second-language assignments between the plurality of second-language speech sounds and the plurality of second-language phonemes; and

determining a mapping configured to associate the plurality of first-language speech sounds with the plurality of second-language speech sounds, wherein determining the cross-lingual assignments is based on the mapping and the second-language assignments.

20. The non-transitory computer readable medium of claim 19, wherein determining the mapping comprises:

determining a first vector that includes representations of first acoustic features in the plurality of first-language speech sounds, and a second vector that includes representations of second acoustic features in the plurality of second-language speech sounds;

determining a third vector that includes association probabilities between the first vector and the second vector; and

determining the mapping based on the third vector.