Identifying language of origin for words using estimates of normalized appearance frequency

• US 20080059151A1
• Filed: 09/01/2006
• Published: 03/06/2008
• Est. Priority Date: 09/01/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Soong, Frank Kao-Ping, You, Jiali, Chu, Min, Chen, Yi Ning
• Granted Patent: US 7,689,408 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

The language of origin of a word or named entity is predicted using estimates of frequency of occurrence of the word or named entity in different languages. In one embodiment, the normalized frequency of occurrence of the word or named entity in a variety of different languages is estimated and the values are used as features in a feature vector which is scored and used to identify language of origin.

Claims

1. A method of identifying a language of origin of an input word, comprising:

generating a wide area network query based on the input word to obtain search results in a plurality of different languages;

estimating a frequency of occurrence of the input word in each of the different languages based on the search results;

identifying the language of origin of the input word based on the estimated frequencies of occurrence, and

outputting an indication of the language of origin.

2. The method of claim 1 wherein identifying the language of origin comprises:

generating a feature vector indicative of the frequencies of occurrence of the input word, in the different languages; and

identifying the language of origin based on the feature vector.

3. The method of claim 1 wherein estimating a frequency of occurrences comprises:

estimating a normalized frequency of occurrence of the input word in each of the different languages.

4. The method of claim 3 wherein the search results comprise web pages and wherein estimating a normalized frequency of occurrence in a selected language comprises:

obtaining a count of a number of web pages in the selected language in the search results that contain the input word; and

estimating a total number of web pages in the selected language.

5. The method of claim 4 wherein estimating a normalized frequency of occurrence in the selected language comprises:

estimating the normalized frequency of occurrence as a ratio the count of the number of web pages that contain the input word to the total number of web pages.

6. The method of claim 4 wherein estimating a total number of web pages in the selected language comprises:

generating a wide area network query based on one or more function words in the selected language to obtain function word search results; and

estimating the total number of web pages based on the function word search result.

7. The method of claim 6 wherein the function word search results comprise an indication of web pages containing the one or more function words and wherein estimating the total number of web pages based on the function word search result, comprises:

estimating the total number of web pages in the selected language as a number of web pages that contain one of the function words.

8. The method of claim 6 wherein generating a wide area network query based on the one or more function words comprises:

obtaining a predefined set of a plurality of function words; and

generating the query using the predefined set of function words.

9. The method of claim 1 and further comprising:

generating an indication of how likely the input word is to have a given language of origin based on a morphological structure of the input word.

10. The method of claim 2 wherein generating the feature vector comprises:

generating morphological structure features indicative of n-gram scores for the input word given each of the different languages; and

generating the feature vector to indicate the morphological structure features.

11. A system for identifying a language of origin of an input word, comprising:

a frequency of occurrence estimation system configured to estimate a frequency of occurrence of the input word in each of a plurality of different languages; and

a language identifier configured to identify the language of origin of the input word based on the frequency of occurrence estimated.

12. The system of claim 11 and further comprising:

a feature extraction system including the frequency of occurrence estimation system; and

a search engine coupled to the feature extraction system, the feature extraction system configured to generate a query based on the input word and extract features from search result, the features being indicative of the frequency of occurrence of the input word.

13. The system of claim 12 wherein the feature extraction component includes:

a morphological feature extractor configured to extract morphological features indicative of how likely the input word has a given language of origin based on a morphological structure of the input word.

14. The system of claim 13 wherein the morphological feature extractor is configured to extract n-gram scores for the input word in each language.

15. The system of claim 14 wherein the language identifier comprises a classifier.

16. The system of claim 12 wherein the feature extraction system extracts normalized frequency of occurrence features based on a number of pages in the search result, in a given language, that contain the input word and an estimate of a total number of pages in the given language.

17. A computer readable medium storing computer readable instructions which, when executed by a computer cause the computer to implement a feature extraction system comprising:

a normalized frequency of occurrence estimation system configured to receive an input word, access contents over a network and extract frequency of occurrence features indicative of a normalized frequency of occurrence of the input word in a plurality of different languages, the frequency of occurrence features being used to identify a language of origin of the input word.

18. The computer readable medium of claim 17 wherein the feature extraction system further comprises:

a morphological feature extractor configured to extract morphological features indicative of a likely language of origin for the input word based on a morphological structure of the input word.

19. The computer readable medium of claim 17 wherein the feature extraction system includes a search engine configured to return results in response to a query based on the input word, the results being indicative of the frequency of occurrence features.