SYSTEM AND METHOD FOR BUILDING DIVERSE LANGUAGE MODELS

• US 20120232885A1
• Filed: 03/08/2011
• Published: 09/13/2012
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: BANGALORE, Srinivas, BARBOSA, Luciano De Andrade
• Granted Patent: US 9,081,760 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 2015/0633 using lexical or orthographic knowledge sources

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for collecting web data in order to create diverse language models. A system configured to practice the method first crawls, such as via a crawler operating on a computing device, a set of documents in a network of interconnected devices according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from previous crawling cycles by crawling documents whose vocabulary considered likely to fill gaps in the current language model. A language model from a previous cycle can be used to guide the creation of a language model in the following cycle. The novelty regions can include documents with high perplexity values over the current language model.

Claims

1. A method of generating a diverse language model, the method comprising:

crawling, via a crawler operating on a computing device, a plurality of documents in a network of interconnected documents according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from at least one previous crawling cycle, and wherein the visitation policy is based on a vocabulary considered likely to fill gaps in the current language model; and

generating a diverse language model based on the current language model and the plurality of documents.

2. The method of claim 1, further comprising recognizing received speech with the diverse language model.

3. The method of claim 1, wherein the diverse language model is generated by modifying the current language model.

4. The method of claim 1, wherein the novelty regions are identified using an information theoretic measure.

5. The method of claim 1, wherein the novelty regions comprise documents with high perplexity values over the current language model from the at least one previous cycle.

6. The method of claim 1, further comprising updating the visitation policy for the crawler once a specified number of pages is crawled.

7. The method of claim 6, wherein updating the visitation policy is based on an expected perplexity value of documents.

8. The method of claim 7, wherein the expected perplexity value of a page is determined by evaluating links to the page.

9. The method of claim 1, further comprising merging a set of language models to yield a combined language model.

10. The method of claim 1, wherein the plurality of documents in a network of interconnected devices comprises web pages.

11. A system for recognizing speech, the system comprising:

a processor;

a first module configured to control the processor to receive speech;

a second module configured to control the processor to load a language model, wherein the language model is generated by crawling a plurality of documents in a network of interconnected documents according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from at least one previous crawling cycle, and wherein the visitation policy is based on a vocabulary considered likely to fill gaps in the current language model; and

a third module configured to control the processor to recognize the speech using the language model.

12. The system of claim 11, wherein the novelty regions are identified using an information theoretic measure.

13. The system of claim 11, wherein the novelty regions comprise documents with high perplexity values over the current language model from the at least one previous cycle.

14. The system of claim 11, wherein the language model is further generated by updating the visitation policy for the crawler once a specified number of pages is crawled.

15. The system of claim 14, wherein updating the visitation policy is based on an expected perplexity value of documents.

16. A non-transitory computer-readable storage medium storing instructions which, when executed by a computing device, cause the computing device to generate a diverse language model, the instructions comprising:

crawling a plurality of documents in a network of interconnected documents according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from at least one previous crawling cycle, and wherein the visitation policy is based on a vocabulary considered likely to fill gaps in the current language model; and

generating a new language model based on the current language model and the plurality of documents.

17. The non-transitory computer-readable storage medium of claim 16, the instructions further comprising recognizing received speech with the new language model.

18. The non-transitory computer-readable storage medium of claim 16, the instructions further comprising updating the visitation policy for the software crawler once a crawling threshold is reached, wherein updating the visitation policy is based on an expected perplexity value of documents, and wherein the expected perplexity value of a page is determined by evaluating links to the page.

19. The non-transitory computer-readable storage medium of claim 16, the instructions further comprising merging a set of language models to yield a combined language model.

20. The non-transitory computer-readable storage medium of claim 16, wherein the second language model comprises a trigram model built using a language modeling toolkit.