Classification filter for processing data for creating a language model

• US 20060178869A1
• Filed: 02/10/2005
• Published: 08/10/2006
• Est. Priority Date: 02/10/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mahajan, Milind V., Mau, Peter K.L., Yu, Dong, Odell, Julian J., Acero, Alejandro
• Granted Patent: US 8,165,870 B2
• Time in Patent Office: Days
• US Class Current: 704/10
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

The method and apparatus utilize a filter to remove a variety of non-dictated words from data based on probability and improve the effectiveness of creating a language model.

Claims

1. A computer-implemented method of processing data for creating a language model, the method comprising:

receiving data which is suitable for creating a language model and data which is not suitable for creating a language model;

processing the data using a classifier to identify at least the data suitable for creating the language model; and

outputting the data suitable for creating the language model.

2. The computer-implemented method of claim 1 wherein processing includes using a classification model which uses a linear or non-linear combination of features.

3. The computer-implemented method of claim 2 wherein processing includes using a linear classifier based classification model.

4. The computer-implemented method of claim 2 wherein processing includes computing a feature of the classification model for units of the data that is independent of normalization of the data and independent of a vocabulary.

5. The computer-implemented method of claim 4 wherein the feature comprises at least one of a number of tokens in the unit, and an average number of characters per token in the unit.

6. The computer-implemented method of claim 2 wherein processing includes normalizing the data to provide words that would be spoken for the written format, and computing a feature of the classification model related to normalization of the data.

7. The computer-implemented method of claim 6 wherein the feature comprises at least one of a percentage of tokens in the data that requires normalization, and a ratio of the number of tokens in a unit of the data to the number of words in the corresponding normalized unit.

8. The computer-implemented method of claim 2 wherein processing includes normalizing the data to provide words that would be spoken for the written format, and computing a feature of the classification model related to comparison to a lexicon.

9. The computer-implemented method of claim 8 wherein the feature comprises at least one of the percentage of words in the normalized text which are end-of-sentence words, and the percentage of words in the normalized text which are not in the lexicon.

10. The computer-implemented method of claim 2 wherein processing includes normalizing the data to provide words that would be spoken for the written format, and computing a feature of the classification model related to comparison to a task independent language model.

11. The computer-implemented method of claim 10 wherein the feature comprises at least one of the perplexity of the normalized text using the task independent language model, the percentage of trigrams in the normalized text which are present in the task independent language model, and the percentage of bigrams in the normalized text which are present in the task independent language model.

12. The computer-implemented method of claim 2 wherein processing includes computing a feature of the classification model of the data wherein the feature comprises a set related to specific ranges.

13. A computer-readable medium having computer-executable instructions for performing steps to process data for creating a language model, the steps comprising:

receiving data which is suitable for creating a language model and data which is not suitable for creating the language model;

dividing the data into a sequence of text units; and

ascertaining based on a probability each text unit that is suitable for creating the language model.

14. The computer-readable medium of claim 13 wherein ascertaining includes using a classification model which uses a linear or non-linear combination of features.

15. The computer-readable medium of claim 14 wherein ascertaining includes using a linear classifier based classification model.

16. The computer-readable medium of claim 14 wherein each text unit comprises a plurality of words.

17. The computer-readable medium of claim 16 wherein each text unit comprises a line of text.

18. The computer-readable medium of claim 14 and further comprising normalizing the data to provide words that would be spoken for the written format therein.

19. The computer-readable medium of claim 18 wherein ascertaining includes computing a feature of the classification model for units of the data that is independent of normalization of the data and independent of a vocabulary.

20. The computer-readable medium of claim 19 wherein ascertaining includes computing a feature of the classification model for units of the data that is based on at least one of normalization of the data and comparison to a vocabulary.