Method and system for language identification

• US 20060229865A1
• Filed: 07/26/2005
• Published: 10/12/2006
• Est. Priority Date: 04/07/2005
• Status: Active Grant

• Current Assignee: LinkedIn Corporation (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Troussov, Alexandre, Nevidomski, Alexei, McCloskey, Daniel, O'Donovan, Brian, Volkov, Pavel, Takeuchi, Mayo, Carlgren, Richard
• Granted Patent: US 7,818,165 B2
• Time in Patent Office: Days
• US Class Current: 704/8
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/268 Morphological analysis
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

A method and system for language identification are provided. The system includes a feature set of a plurality of character strings of varying length with associated information. The associated information includes one or more significance scores for a character string for one or more of a plurality of languages. Means are provided for detecting character strings from the feature set within a token from an input text. The system uses a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string. The associated information can also include significance scores based on linguistic rules.

Claims

1. A system for language identification, comprising:

a feature set of a plurality of character strings of varying length with associated information;

the associated information including one or more significance scores for a character string for one or more of a plurality of languages;

means for detecting character strings from the feature set within a token from an input text.

2. The system of claim 1, wherein the system includes a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string.

3. The system of claim 2, wherein the character strings are compiled as a letter tree minimized to reuse common postfixes with common glosses.

4. The system of claim 1, wherein the significance scores for the plurality of languages include positive and negative scores.

5. The system of claim 1, wherein the associated information includes significance scores based on linguistic rules.

6. The system of claim 1, wherein the associated information includes positional information of the character string within a token.

7. The system of claim 6, wherein the positional information includes complete word, word-initial, word-final, and word-internal indications.

8. The system of claim 1, wherein the feature set is formed of character strings for a plurality of languages, using single or multiple character encodings.

9. The system of claim 1, wherein the system includes means for dividing an input text into lexical or morphological tokens.

10. The system of claim 1, wherein the significance scores for a character string are based on one or more of the group of:

the presence of language specific characters;

function words;

the length of the character string;

the importance of the character string in a training corpus.

11. The system of claim 1, further comprising means for determining the language of the input text from the significance scores.

12. A method for language identification, comprising:

inputting a text;

dividing the text into tokens;

detecting character strings within a token from a feature set of a plurality of character strings of varying length with associated information, the associated information including one or more significance scores for a character string for one or more of a plurality of languages.

13. The method of claim 12, further comprising identifying all character strings within an input text and adding the significance scores for a particular language for all character strings identified.

14. The method of claim 13, further comprising determining the language of the input text by sorting the significance scores for the plurality of languages and selecting the language with the highest score.

15. The method of claim 12, wherein the feature set is provided as a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string.

16. The method of claim 12, wherein the significance scores for the plurality of languages include positive and negative scores.

17. The method of claim 12, wherein the associated information includes significance scores based on linguistic rules.

18. The method of claim 12, wherein the associated information includes positional information of the character string within a token.

19. The method of claim 18, wherein the positional information includes complete word, word-initial, word-final, and word-internal indications.

20. The method of claim 12, wherein the feature set is formed of character strings for a plurality of languages, using single or multiple character encodings.

21. The method of claim 12, wherein the detecting character strings within a token comprises analyzing positional data and looking up a token for decreasing lengths of prefixes.

22. The method of claim 21, wherein word-final tokens and character strings are reversed.

23. A computer program product having a computer readable storage medium, the computer readable storage medium having program code stored thereon for language identification, the program code comprising:

program code for inputting a text;

program code for dividing the text into tokens; and

program code for detecting character strings within a token from a feature set of a plurality of character strings of varying length with associated information, the associated information including one or more significance scores for a character string for one or more of a plurality of languages.

24. A system for compiling a feature set, comprising:

means for compiling a plurality of character strings of varying length;

means for associating information with a character string, including means for allocating one or more significance scores for one or more of a plurality of languages.

25. A method for compiling a feature set, comprising:

compiling a plurality of character strings of varying length; and

associating information with a character string, including allocating one or more significance scores for one or more of a plurality of languages.

26. The method of claim 25, wherein the step of allocating one or more significance scores uses statistical analysis of language texts.

27. The method of claim 25, wherein language models are used to compile the plurality of character strings that are a characteristic feature of a subset of words that should be recognized in a language.

28. A computer program product having a computer readable storage medium, the computer readable storage medium having stored thereon program code for compiling a feature set, the program code comprising:

program code for compiling a plurality of character strings of varying length; and

program code for associating information with a character string, including allocating one or more significance scores for one or more of a plurality of languages.

29. A computer data signal embodied in a carrier wave, the computer data signal including program code for language identification, the program code comprising:

program code for inputting a text;

program code for dividing the text into tokens; and

program code for detecting character strings within a token from a feature set of a plurality of character strings of varying length with associated information, the associated information including one or more significance scores for a character string for one or more of a plurality of languages.

30. A computer data signal embodied in a carrier wave, the computer data signal including code for compiling a feature set, the program code comprising:

program code for compiling a plurality of character strings of varying length; and

program code for associating information with a character string, including allocating one or more significance scores for one or more of a plurality of languages.