Automatic spoken language identification based on phoneme sequence patterns

• US 8,190,420 B2
• Filed: 08/04/2009
• Issued: 05/29/2012
• Est. Priority Date: 08/04/2009
• Status: Active Grant

• Current Assignee: Longsand Limited (Open Text Corporation)
• Original Assignee: Autonomy Corporation PLC (HP Inc.)
• Inventors: Kadirkamanathan, Mahapathy, Waple, Christopher John
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US12/535,038
• Publication Number:
  US 20110035219A1
• Time in Patent Office: 1,029 Days
• Field of Search: 704 2- 8
• US Class Current: 704/8
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams

Abstract

A language identification system that includes a universal phoneme decoder (UPD) is described. The UPD contains a universal phoneme set representing both 1) all phonemes occurring in the set of two or more spoken languages, and 2) captures phoneme correspondences across languages, such that a set of unique phoneme patterns and probabilities are calculated in order to identify a most likely phoneme occurring each time in the audio files in the set of two or more potential languages in which the UPD was trained on. Each statistical language models (SLM) uses the set of unique phoneme patterns created for each language in the set to distinguish between spoken human languages in the set of languages. The run-time language identifier module identifies a particular human language being spoken by utilizing the linguistic probabilities supplied by the one or more SLMs that are based on the set of unique phoneme patterns created for each language.

Claims

1. A language identification engine, comprising:

a front end module having an input configured to receive an audio stream that includes a spoken language that corresponds to at least one of a set of two or more candidate languages being spoken in the audio stream under analysis;

a universal phoneme decoder that contains a universal phoneme set that

     1) represents all phonemes occurring in the set of two or more candidate languages, and

     2) captures phoneme correspondences across languages, such that a set of unique phoneme patterns and probabilities are calculated in order to identify a most likely phoneme occurring each time in the audio stream in the set of two or more candidate languages;

one or more statistical language models having logic configured to supply to a run-time language identifier module probabilities of how linguistically likely a particular uttered phoneme identified by the universal phoneme decoder comes from a particular candidate language based on an identified sequence of phonemes, wherein each statistical language model uses linguistic features from the identified phonemes from the universal phoneme decoder including the set of unique phoneme patterns created for each language in the set to distinguish between spoken human languages in the set of two or more candidate languages;

a bank of two or more human language specific databases for the one or more statistical language models to reference, wherein the databases are pre-filled with phoneme and phoneme sequences for a particular language in the set of two or more candidate languages, and each of the human language specific databases received the phoneme and phoneme sequences from a phoneme output from the universal phoneme decoder independent of which spoken language in the set of two or more candidate languages was being trained on;

wherein the run-time language identifier module identifies a particular human language being spoken in the received audio stream from the set of two or more candidate languages by utilizing the one or more statistical language models; and

wherein the modules making up the language identification engine are implemented in logic blocks of electronic hardware, software coding, and any combination of the two, where portions implemented in software coding are stored in an executable format by a processor on a non-transitory machine-readable medium.

2. The language identification engine of claim 1, wherein the run-time language identifier module queries the one or more statistical language models cooperating with the human language specific databases filled in the training process to observe enough phoneme sequences that correspond to spoken audio so that the run-time language identifier is able to identify the candidate language by utilizing the one or more statistical models, and the training on each human language occurred one language at time to fill each corresponding human language specific database to maximize an accuracy of both a per-language recognition accuracy in identifying a correct phoneme being spoken in that language as well as a language identification process of which candidate language is being spoken;

and wherein the logic block of electronic hardware includes the processor and a memory.

3. The language identification engine of claim 1, wherein the universal phoneme decoder during a training phase identifies the most likely phoneme sequence in the audio stream for each of the languages being trained on, a language ID trainer puts phones and phone sequences into its own associated human language specific database for that spoken language being trained on, and the phone output of the phoneme decoder supplies each identified phoneme in the series of identified phonemes to an input of the language ID trainer.

4. The language identification engine of claim 1, wherein the universal phoneme decoder during the training phase converted uttered sounds in supplied audio training data into identified phonemes in each candidate language, a language ID trainer coupled to the universal phoneme decoder analyzed the resulting phonemes and the phoneme sequences, and then the language ID trainer filled the human language specific databases for the one or more statistical language models for each candidate language on a per language basis.

5. The language identification engine of claim 1, wherein the statistical language models accumulate a set of n-gram sequences of phonemes histograms, one per language, in an assumption that different languages will have different n-gram histograms, and a language ID trainer then approximates the n-gram distribution as the weighted sum of the probabilities of the n-gram sequence of phonemes and supplies this back to the statistical language model for that language.

6. The language identification engine of claim 1, wherein the statistical language model compares both a ratios of counts of phone sequences observed in an audio training data compared to 1) how often particular phonemes and phoneme sequences are used in a first human language to an occurrence of other phoneme and phoneme sequences in the first human language, and 2) how often particular phonemes and phoneme sequences are used in the first human language to an occurrence of the same or very similar sounding phonemes and phoneme sequences are used in a second human language.

7. The language identification engine of claim 1, wherein the universal phoneme set includes unique phonemes and/or phoneme sequences that only occur in one or a few languages, and on the other end of the spectrum, the unique phoneme and/or phoneme sequence occurs with such a high occurrence rate in a particular language compared to other languages in the set that the occurrence of this phoneme and/or phoneme sequence accompanied by multiple occurrences of this phoneme occurring within a short set time period is also a good indicator at identifying that a particular language is being spoken.

8. The language identification engine of claim 1, wherein the identified phonemes and the phoneme sequences from the language ID trainer in the training phase are modeled in the one or more statistical language models using discrete Markov models (HMMs) and a same universal phoneme decoder was applied to each candidate language in the set of languages, as opposed to a phoneme decoder being specifically trained to the candidate language being tested, which now allows a more consistent output from the universal phoneme decoder for the received audio stream.

9. The language identification engine of claim 1, wherein each statistical model has its own human language specific database full of phones and phone sequences for that spoken language, each statistical model analyzes an amount of different phones and phone sequences that occur in a training audio data and counts of a total number of phonemes for the training audio data upon which the model is based on, each statistical language model uses the linguistic features including the set of unique phoneme patterns to distinguish between spoken human languages, each statistical language model couples to the run-time language identification and supplies to the language identifier module probabilities of how linguistically likely a particular uttered phoneme comes from a particular spoken language based on the identified sequence of a phonemes.

10. The language identification engine of claim 1, wherein each human language specific database couples to a language ID trainer module, each human language specific database acts as a repository for that language to store language ID parameters including all special N-grams, sequences of linguistic items, that have significantly different counts/occurrences in a corpus of human language specific acoustic data analyzed than would be expected compared to other languages, and the special N-grams are stored along with an actual counts of the number of times that N-gram appeared in the corpus of human language specific acoustic data analyzed, and a first human language specific database during run time couples to a first statistical language model.

11. The language identification engine of claim 1, wherein during the run-time language identification phase, language ID parameters for each language to be identified are loaded into the run-time language identifier module, a new utterance is compared to each of the language-dependent statistical models, and the likelihood that the language of the utterance matches the languages used to train the models is calculated by the run-time language identifier module, and a language-dependent statistical language model determined by the run-time language identifier module most likely to be correct is then selected by the run-time language identifier module.

12. The language identification engine of claim 1, wherein the front-end converts the received audio stream into time coded feature frames for language identification and the universal phoneme decoder recognizes the feature frames as a sequence of phonemes, together with start/end time associated with each feature frame, and the universal phoneme detector is configured to identify all of the phonemes uttered in each of the set of languages to be identified, and the run-time language identifier module receives the phoneme sequence from the universal phoneme decoder in the time coded feature frames and determines a most probable candidate language based on the language identifying algorithm making use of the set of unique phoneme patterns to a given spoken language verses the common phoneme sequences across the different languages.

13. A continuous speech recognition engine hosted on a server cooperating with a language identification engine that improves an accuracy of probability estimates, comprising:

an input to receive supplied audio files from a client machine over a wide area network to the server hosting the continuous speech recognition engine;

a user interface to the continuous speech recognition engine; and

a language identification engine, wherein the continuous speech recognition engine receives an input from the language identification engine to identify the human language being spoken in the received audio files;

wherein the language identification engine at least includes

a front end module having an input configured to receive an audio files consisting of a spoken language of at least one of a set of two or more potential languages being spoken in the audio files under analysis,

a universal phoneme decoder that contains a universal phoneme set representing both

     1) all phonemes occurring in the set of two or more spoken languages, and

     2) captures phoneme correspondences across languages, such that a set of unique phoneme patterns and probabilities are calculated in order to identify a most likely phoneme occurring each time in the audio files in the set of two or more potential languages in which the universal phoneme decoder was trained on,

one or more statistical language models having logic configured to supply to a run-time language identifier module probabilities of how linguistically likely a particular uttered phoneme identified by the universal phoneme decoder comes from a particular spoken language based on an identified sequence of phonemes, wherein each statistical language model uses linguistic features from the identified phonemes from the universal phoneme decoder including the set of unique phoneme patterns created for each language in the set to distinguish between spoken human languages in the set of two or more spoken languages,

a bank of human language specific databases for the one or more statistical language models to reference, where of the databases were filled with phoneme and phoneme sequences being trained on for a particular language in the set of two or more spoken languages, and each of the human language specific databases received the phoneme and phoneme sequences from a phoneme output from the same universal phoneme decoder independent of which spoken language in the set of two or more potential languages was being trained on, and

the run-time language identifier module identifies a particular human language being spoken in the received audio files in the set of two or more potential languages by utilizing the linguistic probabilities supplied by the one or more statistical language models that are based on the set of unique phoneme patterns created for each language, wherein the modules making up the language identification engine are implemented in logic blocks of electronic hardware, software coding, and any combination of the two, where portions implemented in software coding are stored on a non-transitory machine-readable medium.

14. The continuous speech recognition engine hosted on the server of claim 13, wherein a language identification algorithm in the run-time language identifier module is a second order discrete Markov model with a dialogue structure and branch logic and the language identification algorithm in the run-time language identifier module uses the second order Markov Model algorithm based on the set of unique phoneme and phoneme sequences, and recognition involves tokenizing the audio file, and calculating a likelihood that its phone sequence was produced in each of the languages in the set, and the language is identified using the set of unique phoneme patterns in a single recognition pass through the system.

15. The continuous speech recognition engine hosted on the server of claim 13, further comprising another input to receive query words of interest from a user of the client machine to the user interface to find out if the received audio file contains any of the query words, and an intelligence engine identifies recognized words from the query and returns a hierarchical rank list of recognized words, and the user then activates a link to the returned time segments in the received audio files containing those recognized words matching the query words and listen to a segment of the audio files pertinent to when those words are spoken in the received audio files.

16. The continuous speech recognition engine hosted on the server of claim 13, further comprising:

a triggering and synchronization module of the continuous speech recognition engine that analyzes call center audio conversations and identifies when certain words of interest are spoken with the triggering and synchronization module, wherein the triggering and synchronization module then directs a user on the client machine to the time segment containing those words matching the trigger words and allows the user to listen to a segment of the audio files pertinent to when those trigger words are spoken in the supplied audio files.

17. The continuous speech recognition engine hosted on the server of claim 16, wherein the triggering and synchronization module sends an event notification to the client machine over the network so the user on the client machine can activate/click on the notification to allow the user to listen to a segment of the audio files pertinent to when those trigger words are spoken in the received audio files.

18. A method of language identification, comprising:

receiving an audio stream consisting of a spoken language of at least one of a set of two or more potential languages being spoken in the audio stream under analysis;

identifying a most likely phoneme occurring each time in the audio stream under analysis in the set of two or more potential languages with a universal phoneme decoder that contains a universal phoneme set representing both

     1) all phonemes occurring in the set of two or more spoken languages, and

     2) captures phoneme correspondences across languages, such that a set of unique phoneme patterns and probabilities are calculated;

supplying probabilities of how linguistically likely a particular uttered phoneme identified by the universal phoneme decoder comes from a particular spoken language based on an identified sequence of phonemes from one or more statistical language models, wherein each statistical language model uses linguistic features from the identified phonemes from the universal phoneme decoder including the set of unique phoneme patterns created for each language in the set to distinguish between spoken human languages in the set of two or more spoken languages;

referencing a first human language specific database with a first statistical language models, where of the first human language specific database was filled with phoneme and phoneme sequences being trained on for a particular language in the set of two or more spoken languages during a training phase, and each of the human language specific databases that includes the first human language specific database received the phoneme and phoneme sequences from a phone output from the same universal phoneme decoder independent of which spoken language in the set of two or more potential languages was being trained on, however, each human language specific database was filled one at a time on a per specific language basis; and

at run-time, identifying a particular human language being spoken in the received audio stream in the set of two or more potential languages by utilizing the linguistic probabilities supplied by the one or more statistical language models that are based on the set of unique phoneme patterns created for each language.

19. The method of claim 18, further comprising:

during the run-time language identification phase, language ID parameters for each language to be identified are loaded into a run-time language identifier module;

during the run-time language identification phase, a new utterance is compared to each of the language-dependent statistical models, and the likelihood that the language of the utterance matches the languages used to train the models is calculated by the run-time language identifier module; and

a language-dependent statistical language model determined by the run-time language identifier module most likely to be correct is then selected by the run-time language identifier module.

20. The method of claim 19, wherein the run-time language identifier module queries the one or more statistical language models cooperating with the human language specific databases filled in the training process to observe enough phoneme sequences that correspond to spoken audio so that the run-time language identifier is able to identify the spoken language by utilizing the one or more statistical models, and training on each human language occurred one language at a time to fill each corresponding human language specific database to maximize an accuracy of both a per-language recognition accuracy in identifying a correct phoneme being spoken in that language as well as a language identification process of which language is being spoken.

21. The method of claim 19, further comprising:

identifying each phoneme with the universal phoneme decoder to identify the language being spoken from an audio stream of a live broadcast as well as detecting and identifying a new language being spoken within the same audio stream;

supplying the identified language and identified phonemes to a speech recognition module, and subsequently identifying each word in the identified language with the speech recognition module from the audio stream of the live broadcast;

encoding each of the identified phonemes and identified words from the audio stream of the live broadcast;

assigning a time indication with each of the identified words, where each of the identified words shares a common time reference;

generating a synchronized link to relevant material based on the content of the live broadcast, the synchronized link to be displayed with the live broadcast; and

synchronizing the synchronized link to appear at approximately an utterance of the most relevant word related to the content of the live broadcast, wherein the link to relevant material based is generated upon the one or more words being spoken and synchronizing a display of the link in less than ten seconds from analyzing the audio stream of the live broadcast.