Method of multilingual speech recognition by reduction to single-language recognizer engine components

• US 7,689,404 B2
• Filed: 02/16/2005
• Issued: 03/30/2010
• Est. Priority Date: 02/24/2004
• Status: Expired due to Fees

• Current Assignee: Arkady Khasin
• Original Assignee: Arkady Khasin
• Inventors: Khasin, Arkady
• Primary Examiner(s): Vo, Huyen X.
• Application Number:
  US10/906,376
• Publication Number:
  US 20050187758A1
• Time in Patent Office: 1,868 Days
• Field of Search: 704/255, 704/277, 704/256, 704/231, 704/251, 704/254, 704/1, 704/2, 704/9, 704/10, 704/238, 704/242, 704/240, 704/256.4, 704/256.5, 704/3, 704/235
• US Class Current: 704/1
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence

Abstract

In some speech recognition applications, not only the language of the utterance is not known in advance, but also a single utterance may contain words in more than one language. At the same time, it is impractical to build speech recognizers for all expected combinations of languages. Moreover, business needs may require a new combination of languages to be supported in short period of time. The invention addresses this issue by a novel way of combining and controlling the components of the single-language speech recognizers to produce multilingual speech recognition functionality capable of recognizing multilingual utterances at a modest increase of computational complexity.

Claims

1. A method of speech recognition comprising:

providing a multilingual dispatcher engine in a computing system having at least a processor, a memory, text and audio input devices and a text input device;

providing individual components of language-specific single-language speech recognizers in the computing system; and

integrating the multilingual dispatcher engine in the computing system with language-independent functions and executing end-of-utterance detection and coordinating execution of the single-language recognizer components, the single-language recognizer components including at least

a first component executed by the computing system to produce one or more language-specific word models for any given word in a recognition vocabulary,

a second component executed by the computing system to compute a language-specific speech feature vector from a slice of acoustic data,

a third component executed by the computing system to update a state of a word hypothesis by the latest computed feature vector, a language-specific pronunciation, and a facility to initialize a state,

and a fourth component executed by the computing system to update a state of a word hypothesis by an exit state of any state of any word hypothesis as immediately preceding segment of an utterance, the fourth component referred to as seeding, the seeding comprising computing the entry state of a pronunciation'"'"'s model by iterating over the exit states supplied to the seeding component,

wherein coordination of execution comprises at least dispatching a step of recognition of a slice of speech, obtaining the scores of word hypotheses, cross-language seeding of word hypothesis and recovering the winning multilingual text from the winning word hypothesis at end of utterance.

2. The method of claim 1 wherein the multilingual dispatcher engine maintains a recognition vocabulary in which pronunciations of words in different languages are produced by executing corresponding language-specific components and storing the pronunciations in that said language'"'"'s component format along with a means of recovering the producing language.

3. The method of claim 2 wherein the multilingual dispatcher engine maintains and updates runtime states of language-specific pronunciations by executing the language-specific state initialization and state update components against the language-specific format of a current feature vector.

4. The method of claim 3 wherein the multilingual dispatcher engine includes score scaling functions, which convert a hypothesis score from its language-specific scale to a multilingual dispatcher engine scale and from the multilingual dispatcher engine scale to a language-specific scale, these functions enabled as best-fit functions to generate equal multilingual dispatcher engine scores for silence and, optionally, noise hypotheses, in different languages on qualified application-dependent noise utterances.

5. The method of claim 4 wherein the language-specific recognizers include a component computing a score adjustment for switching to another language after a given pronunciation and a component computing a score adjustment for switching from another language before a given pronunciation.

6. The method of claim 5 wherein the multilingual dispatcher engine includes language-specific tunable parameters referred to as language switch penalties, one per pair of languages, for score adjustment when switching from and to a given language.

7. The method of claim 6 wherein the multilingual dispatcher engine updates the initial state of a pronunciation Q from an exit state of a pronunciation P in the same language by executing the corresponding language-specific component.

8. The method of claim 7 wherein the multilingual dispatcher engine includes updating the initial state of a pronunciation Q from an exit state L of a pronunciation P in a different language, the updating comprising:

creating in the computing system a temporary exit state by cloning the state L and converting its score to the multilingual dispatcher engine format;

determining in the computing system out-of-language of P score adjustment by executing the corresponding component of the language of P, if available, and converting it to the multilingual dispatcher engine format;

determining in the computing system into-language of Q score adjustment by executing the corresponding component of the language of Q, if available, and converting it to the multilingual dispatcher engine format;

adjusting in the computing system the score of the temporary exit state by adding a weighted average of out-of language and into-language score adjustments computed by the corresponding language-specific components when available;

adjusting in the computing system the score of the temporary exit state by adding the language switch penalty;

converting in the computing system the score of the temporary exit state into the format of the language of Q; and

executing in the computing system the language-specific seeding component of the language of Q with the temporary exit state.

9. The method of claim 8 wherein the temporary exit state is not explicitly produced unless computationally efficient.

10. The method of claim 9 wherein the language specific recognizers comprise a facility to prune active states of pronunciations based on score threshold and the multilingual dispatcher engine controlling the pruning threshold of each individual recognizer, the controlling comprising:

determining the common prune threshold as the minimum of individual thresholds produced by said language specific recognizers, scaled to the multilingual dispatcher engine scale, relaxed by a tunable parameter; and

dispatching a common pruning threshold to each said recognizer in the format scaled to the format of that recognizer.

11. The method of claim 10 wherein components common to all supported languages are factored out of language-specific recognizers and incorporated into the multilingual dispatcher engine.

12. The method of claim 1 wherein the computing system is embedded in a hand-held interactive device.

13. A method of speech recognition comprising:

in a memory of a computing system having a processor, text/audio input devices and a text output device, providing a multilingual dispatcher component and individual components of language-specific single-language speech recognizers in a memory executed by the computing system, the multilingual dispatcher component integrating language-independent functions and executing in parallel an end-of-utterance detection process and speech recognition process, the speech recognition process comprising;

processing a slice of acoustic data into a language-independent feature vector in the multilingual dispatcher component, dispatching the feature vector to each active single-language recognizer and receiving the language-specific feature vector from each recognizer;

dispatching in the multilingual dispatcher component the language-specific feature vectors each to its corresponding single-language recognizer to execute state update of word hypotheses by the corresponding recognizer;

obtaining in the multilingual dispatcher component word hypotheses numeric scores from each single-language recognizer and scaling them to an internal scale of the multilingual dispatcher component;

determining in the multilingual dispatcher component of the computing system a resulting numeric score of each word hypothesis as a least scaled score of a hypothesis among all single-language recognizers and, for the winning word hypothesis of each word, remembering the language that produced it;

dispatching in the multilingual dispatcher component the execution of updating each word hypothesis using all winning word hypotheses within, the updating referred to as seeding, and using each of the single-language recognizer components, wherein;

if the seeding state with the winning multilingual dispatcher component score belongs to the same language as the language of the recognizer then the built-in update facility of the recognizer is executed without modifications on the pair of the seeding and the seeded word hypotheses;

if the seeding state belongs to a different language, applying language transition adjustments to the seeding score in the multilingual dispatcher component format which is then resealed to the scale of the single-language recognizer and replaces the seeding score, after which replacement the single-language recognizer'"'"'s built-in update facility is executed on the pair of the modified seeding and the seeded word hypotheses;

declaring in the multilingual dispatcher component the winning hypothesis at the end of utterance as the utterance ending with the winning word hypothesis, and recovering the words of the utterance along with the language of each word from back trace information associated with seeding.

14. The method of claim 13 wherein each of the language-specific single-language speech recognizers comprises a pronunciation model for the end-of-utterance word and the multilingual dispatcher component monitors end of utterance as the event of the end-of-utterance hypothesis having the overall winning multilingual dispatcher component score.