Creating a language model for a language processing system
First Claim
Patent Images
1. A system including instructions executable by a processor which, when implemented execute a method to build a task dependent unified language model, the method comprising:
- accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts, each of the context-free grammars having words present in a task independent corpus to form the semantic or syntactic concepts;
parsing the task independent corpus with the plurality of context-free grammars to identify word occurrences of each of the semantic or syntactic concepts;
replacing each of the identified word occurrences with corresponding non-terminal tokens;
building a N-gram model having the non-terminal tokens; and
storing the N-gram model and a second plurality of context-free grammars comprising at least some of the same non-terminals representing the same semantic or syntactic concepts, each of the context-free grammars of the second plurality being more appropriate for use in the selected application.
1 Assignment
0 Petitions
Accused Products
Abstract
A method for creating a language model from a task-independent corpus is provided. In one embodiment, a task dependent unified language model is created. The unified language model includes a plurality of context-free grammars having non-terminals and a hybrid N-gram model having at least some of the same non-terminals embedded therein.
-
Citations
17 Claims
-
1. A system including instructions executable by a processor which, when implemented execute a method to build a task dependent unified language model, the method comprising:
-
accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts, each of the context-free grammars having words present in a task independent corpus to form the semantic or syntactic concepts;
parsing the task independent corpus with the plurality of context-free grammars to identify word occurrences of each of the semantic or syntactic concepts;
replacing each of the identified word occurrences with corresponding non-terminal tokens;
building a N-gram model having the non-terminal tokens; and
storing the N-gram model and a second plurality of context-free grammars comprising at least some of the same non-terminals representing the same semantic or syntactic concepts, each of the context-free grammars of the second plurality being more appropriate for use in the selected application. - View Dependent Claims (2, 3, 4, 5)
-
-
6. A system including instructions executable by a processor which, when implemented execute a method to build language model, the method comprising:
-
accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts of the selected application;
generating word phrases from the plurality of context-free grammars;
formulating an information retrieval query from at least one of the word phrases;
querying a task independent corpus based on the query formulated;
identifying associated text in the task independent corpus based on the query; and
building a language model using the identified text. - View Dependent Claims (7, 8, 9, 10, 11, 12, 13, 14)
-
-
15. A system including instructions executable by a processor which, when implemented execute a method to build a unified language model for a selected application, the method comprising:
-
accessing a plurality of context-free grammars comprising non-terminal tokens representing semantic or syntactic concepts of the selected application;
building a word language model from a corpus; and
assigning probabilities to words of at least some of the context-free grammars as a function of corresponding probabilities obtained for the same terminals from the word language model wherein assigning probabilities includes normalizing the probabilities of the words from the word language model in each of the context-free grammars as a function of the words allowed by the corresponding context-free grammar. - View Dependent Claims (16, 17)
-
Specification