System and method for building diverse language models

• US 9,727,557 B2
• Filed: 07/18/2016
• Issued: 08/08/2017
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Barbosa, Luciano De Andrade, Bangalore, Srinivas
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): KOVACEK, DAVID M
• Application Number:
  US15/212,944
• Publication Number:
  US 20160328390A1
• Time in Patent Office: 386 Days
• Field of Search: 704 1, 704 4, 704 8- 10, 704231, 704236, 704243-245, 704251, 704270-2701, 704277, 704E11001-E11007, 706 15- 16, 706 25, 706 45- 61, 706 62
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 2015/0633 using lexical or orthographic knowledge sources

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for collecting web data in order to create diverse language models. A system configured to practice the method first crawls, such as via a crawler operating on a computing device, a set of documents in a network of interconnected devices according to a visitation policy, wherein the visitation policy is configured to focus on novelty regions for a current language model built from previous crawling cycles by crawling documents whose vocabulary considered likely to fill gaps in the current language model. A language model from a previous cycle can be used to guide the creation of a language model in the following cycle. The novelty regions can include documents with high perplexity values over the current language model.

Claims

1. A method comprising:

establishing a crawling schedule configured to identify, according to a pattern of links, a likelihood of web pages to have information capable of filling vocabulary gaps, and wherein a website visitation policy comprises the crawling schedule according to perplexity of the web pages with respect to a language model;

crawling, via a processor, the web-pages based on the crawling schedule, to yield new vocabulary words; and

generating a new language model according to the language model and the new vocabulary words.

2. The method of claim 1, further comprising recognizing received speech with the new language model.

3. The method of claim 1, wherein the new language model is generated by modifying the language model.

4. The method of claim 1, wherein the likelihood of the web pages is further according to an information theoretic measure.

5. The method of claim 4, wherein the web pages have high perplexity values over the language model from a previous cycle.

6. The method of claim 1, further comprising updating a website visitation policy for the crawling once a specified number of pages is crawled.

7. The method of claim 6, wherein updating the website visitation policy is according to an expected perplexity value of novelty regions.

8. The method of claim 7, wherein the expected perplexity value of the novelty regions is determined by evaluating links to the web page.

9. The method of claim 1, further comprising merging a set of language models.

10. A system comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform instructions comprising;

establishing a crawling schedule configured to identify, according to a pattern of links, a likelihood of web pages to have information capable of filling vocabulary gaps, and wherein a website visitation policy comprises the crawling schedule according to perplexity of the web pages with respect to a language model;

crawling, via a processor, the web-pages based on the crawling schedule, to yield new vocabulary words; and

generating a new language model according to the language model and the new vocabulary words.

11. The system of claim 10, the computer-readable storage medium having additional instructions stored which, when executed by the processor, cause the processor to perform operations comprising recognizing received speech with the new language model.

12. The system of claim 10, wherein the new language model is generated by modifying the language model.

13. The system of claim 10, wherein the likelihood of the web pages is further according to an information theoretic measure.

14. The system of claim 13, wherein the web pages have high perplexity values over the language model from a previous cycle.

15. The system of claim 10, the computer-readable storage medium having additional instructions stored which, when executed by the processor, cause the processor to perform operations comprising updating a website visitation policy for the crawling once a specified number of pages is crawled.

16. The system of claim 15, wherein updating the website visitation policy is according to an expected perplexity value of novelty regions.

17. The system of claim 16, wherein the expected perplexity value of the novelty regions is determined by evaluating links to the web page.

18. The system of claim 10, the computer-readable storage medium having additional instructions stored which, when executed by the processor, cause the processor to perform operations comprising merging a set of language models.

19. A non-transitory computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

establishing a crawling schedule configured to identify, according to a pattern of links, a likelihood of web pages to have information capable of filling vocabulary gaps, and wherein a website visitation policy comprises the crawling schedule according to perplexity of the web pages with respect to a language model;

crawling, via a processor, the web-pages based on the crawling schedule, to yield new vocabulary words; and

generating a new language model according to the language model and the new vocabulary words.

20. The non-transitory computer-readable storage device of claim 19, having additional instructions stored which, when executed by the computing device, cause the computing device to perform operations comprising recognizing received speech with the new language model.