Creating a language model for a language processing system

• US 7,286,978 B2
• Filed: 04/11/2006
• Issued: 10/23/2007
• Est. Priority Date: 06/01/2000
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Ye-Yi, Huang, Xuedong D., Mou, Xiaolong, Mahajan, Milind V.
• Primary Examiner(s): Hudspeth; David
• Assistant Examiner(s): Ng; Eunice
• Application Number:
  US11/401,507
• Publication Number:
  US 20060184354A1
• Time in Patent Office: 560 Days
• Field of Search: 704/1, 704/9
• US Class Current: 704/9
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams

Abstract

A method for creating a language model from a task-independent corpus is provided. In one embodiment, a task dependent unified language model is created. The unified language model includes a plurality of context-free grammars having non-terminals and a hybrid N-gram model having at least some of the same non-terminals embedded therein.

Claims

1. A system, comprising:

memory;

a processor adapted to access the memory;

instructions stored on the memory that are executable by the processor which, when implemented, execute a method to build a task dependent unified language model, the method comprising;

accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts, each of the context-free grammars having words present in a task independent corpus to form the semantic or syntactic concepts;

parsing the task independent corpus with the plurality of context-free grammars to identify word occurrences of each of the semantic or syntactic concepts;

replacing each of the identified word occurrences with corresponding non-terminal tokens;

building a N-gram model having the non-terminal tokens; and

storing the N-gram model and a second plurality of context-free grammars comprising at least some of the same non-terminals representing the same semantic or syntactic concepts in the memo, each of the context-free grammars of the second plurality being more appropriate for use in the selected application.

2. The system of claim 1 wherein the plurality of context-free grammars include at least one context-free grammar having a non-terminal token for a phrase that can be mistaken for one of the desired task dependent semantic or syntactic concepts.

3. The system of claim 2 wherein replacing each of the identified word occurrences with corresponding non-terminal tokens includes excluding the non-terminals added for the prevention of mistakes during parsing.

4. The system of claim 2 having instructions further comprising:

storing the N-gram model having the non-terminal tokens and the set of context-free grammars having non-terminal tokens representing task dependent semantic or syntactic concepts on the memory.

5. The system of claim 2 wherein building the N-gram model includes eliminating at least some of the associated text from the task independent corpus for non-terminal tokens that can be mistaken for one of the desired task dependent semantic or syntactic concepts.

6. A system, comprising:

memory;

a processor adapted to access the memory;

instructions stored on the memory and executable by the processor which, when implemented, execute a method to build a language model, the method comprising;

accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts of the selected application;

generating word phases from the plurality of context-free grammars;

formulating an information retrieval query from at least one of the word phases;

querying a task independent corpus based on the query formulated;

identifying associated text in the task independent corpus based on the query; and

building the language model using the identified text for storage on the memory.

7. The system of claim 6 wherein the language model is an N-gram language model.

8. The system of claim 7 and having instructions further comprising:

parsing the identified text of the task independent corpus with the plurality of context-free grammars to identify word occurrences for each of the semantic or syntactic concepts;

replacing each of the identified word occurrences with corresponding non-terminal tokens; and

wherein building the N-gram language model comprises building a N-gram model having the non-terminal tokens.

9. The system of claim 7 and having instructions further comprising:

building a second N-gram language model from the word phases from the plurality of context-free grammars; and

combining the first-mentioned N-gram language model and the second N-gram language model to form a third N-gram language model.

10. The system of claim 9 and having instructions further comprising:

parsing the identified text of the task independent corpus with the plurality of context-free grammars to identify word occurrences for each of the semantic or syntactic concepts;

replacing each of the identified word occurrences with corresponding non-terminal tokens; and

wherein the word phrases include non-terminal tokens and wherein building the first-mentioned N-gram language model comprises building a N-gram model having the non-terminal tokens.

11. The system of claim 8 and having instructions further comprising:

storing the N-gram model having the non-terminal tokens and the plurality of context-free grammars having non-terminal tokens representing task dependent semantic or syntactic concepts on the memory.

12. The system of claim 6 and having instructions further comprising:

storing the identified text of the task independent corpus separate from the task independent corpus.

13. The system of claim 9 wherein building the second N-gram language model includes using only the identified text.

14. The system of claim 13 and having instructions further comprising:

storing the identified text of the task independent corpus separate from the task independent corpus.

15. A system, comprising:

memory;

a processor adapted to access the memory;

instructions stored on the memory and executable by the processor which, when implemented, execute a method to build a unified language model for a selected application, the method comprising;

accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts of the selected application;

building a word language model from a corpus; and

assigning probabilities to words of at least some of the context-free grammars as a function of corresponding probabilities obtained for the same terminals from the word language model wherein assigning probabilities includes normalizing the probabilities of the words from the word language model in each of the context-free grammars as a function of the words allowed by the corresponding context-free grammar.

16. The system of claim 15 wherein the word language model comprises an N-gram language model and wherein the corpus comprises a task independent corpus.

17. The system of claim 16 and having instructions further comprising:

generating word phrases from the plurality of context-free grammars;

formulating an information retrieval query from at least one of the word phrases;

querying the task independent corpus based on the query formulated;

identifying associated text in the task independent corpus based on the query; and

wherein building a N-gram language model includes using the identified text.