Method, Apparatus and Computer Program Product for Providing Flexible Text Based Language Identification

• US 20080147380A1
• Filed: 12/18/2006
• Published: 06/19/2008
• Est. Priority Date: 12/18/2006
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Barliga, Bogdan, Iso-Sipila, Juha, Harju, Mikko A.
• Granted Patent: US 7,552,045 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/263 Language identification

Abstract

An apparatus for providing flexible text based language identification includes an alphabet scoring element, an n-gram frequency element and a processing element. The alphabet scoring element may be configured to receive an entry in a computer readable text format and to calculate an alphabet score of the entry for each of a plurality of languages. The n-gram frequency element may be configured to calculate an n-gram frequency score of the entry for each of the plurality of languages. The processing element may be in communication with the n-gram frequency element and the alphabet scoring element. The processing element may also be configured to determine a language associated with the entry based on a combination of the alphabet score and the n-gram frequency score.

Claims

1. A method comprising:

receiving an entry in a computer readable text format;

determining an alphabet score of the entry for each of a plurality of languages;

determining an n-gram frequency score of the entry for each of the plurality of languages; and

determining a language associated with the entry based on a combination of the alphabet score and the n-gram frequency score.

2. A method according to claim 1, wherein determining the alphabet score comprises comparing characters associated with the entry to an alphabet of each of the plurality of languages and generating a score for each of the plurality of languages, the score for each of the plurality of languages being based at least in part on a failure of one or more of the characters to be represented in the corresponding alphabet of a respective one of the plurality of languages being scored.

3. A method according to claim 1, wherein determining the n-gram frequency score comprises, for each one of the plurality of languages, comparing the entry to n-gram statistics for the one of the plurality of languages.

4. A method according to claim 3, wherein the entry comprises n-characters and wherein comparing the entry to n-gram statistics comprises determining a conditional probability of the nth character of the entry given the preceding n−

1 characters.

5. A method according to claim 3, further comprising assigning a start character and a stop character to a first character and a last character of the entry, respectively, for use in correlation with corresponding start characters and stop characters associated with each n-gram probability of the n-gram statistics.

6. A method according to claim 1, further comprising comparing the alphabet score for each of the plurality of languages to a threshold.

7. A method according to claim 6, wherein determining the n-gram frequency score is only performed for ones of the plurality of languages for which the alphabet score meets the threshold.

8. A method according to claim 6, further comprising assigning the entry a predetermined alphabet score in response to the alphabet score failing to meet the threshold.

9. A method according to claim 1, wherein determining the language associated with the entry comprises selecting the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.

10. A method according to claim 1, wherein the entry is one of a plurality of segments comprising an input text and wherein the determining the language further comprises determining the language associated with the input text based on a cumulative combination of alphabet scores and n-gram frequency scores of each one of the plurality of segments.

11. A computer program product comprising at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for receiving an entry in a computer readable text format;

a second executable portion for determining an alphabet score of the entry for each of a plurality of languages;

a third executable portion for determining an n-gram frequency score of the entry for each of the plurality of languages; and

a fourth executable portion for determining a language associated with the entry based on a combination of the alphabet score and the n-gram frequency score.

12. A computer program product according to claim 11, wherein the second executable portion includes instructions for comparing characters associated with the entry to an alphabet of each of the plurality of languages and generating a score for each of the plurality of languages, the score for each of the plurality of languages being based at least in part on a failure of one or more of the characters to be represented in the corresponding alphabet of a respective one of the plurality of languages being scored.

13. A computer program product according to claim 11, wherein the third executable portion includes instructions for comparing the entry to n-gram statistics for the one of the plurality of languages for each one of the plurality of languages.

14. A computer program product according to claim 13, wherein the entry comprises n-characters and wherein the third executable portion further includes instructions for determining a conditional probability of the nth character of the entry given the preceding n−

1 characters.

15. A computer program product according to claim 13, further comprising a fifth executable portion for assigning a start character and a stop character to a first character and a last character of the entry, respectively, for use in correlation with corresponding start characters and stop characters associated with each n-gram probability of the n-gram statistics.

16. A computer program product according to claim 11, further comprising a fifth executable portion for comparing the alphabet score for each of the plurality of languages to a threshold.

17. A computer program product according to claim 16, wherein the third executable portion is only executed for ones of the plurality of languages for which the alphabet score meets the threshold.

18. A computer program product according to claim 16, further comprising a sixth executable portion for assigning the entry a predetermined alphabet score in response to the alphabet score failing to meet the threshold.

19. A computer program product according to claim 11, wherein the fourth executable portion includes instructions for selecting the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.

20. A computer program product according to claim 11, wherein the entry is one of a plurality of segments comprising an input text and wherein the fourth executable portion includes instructions for determining the language associated with the input text based on a cumulative combination of alphabet scores and n-gram frequency scores of each one of the plurality of segments.

21. An apparatus comprising:

an alphabet scoring element configured to receive an entry in a computer readable text format and calculate an alphabet score of the entry for each of a plurality of languages;

an n-gram frequency element configured to calculate an n-gram frequency score of the entry for each of the plurality of languages; and

a processing element in communication with the n-gram frequency element and the alphabet scoring element, the processing element configured to determine a language associated with the entry based on a combination of the alphabet score and the n-gram frequency score.

22. An apparatus according to claim 21, wherein the alphabet scoring element is further configured to compare characters associated with the entry to an alphabet of each of the plurality of languages and generate a score for each of the plurality of languages, the score for each of the plurality of languages being based at least in part on a failure of one or more of the characters to be represented in the corresponding alphabet of a respective one of the plurality of languages being scored.

23. An apparatus according to claim 21, wherein the n-gram frequency element is further configured to, for each one of the plurality of languages, compare the entry to n-gram statistics for the one of the plurality of languages.

24. An apparatus according to claim 23, wherein the entry comprises n-characters and wherein the n-gram frequency element is further configured to determine a conditional probability of the nth character of the entry given the preceding n−

1 characters.

25. An apparatus according to claim 23, wherein the n-gram frequency element is further configured to assign a start character and a stop character to a first character and a last character of the entry, respectively, for use in correlation with corresponding start characters and stop characters associated with each n-gram probability of the n-gram statistics.

26. An apparatus according to claim 21, the alphabet scoring element is further configured to compare the alphabet score for each of the plurality of languages to a threshold.

27. An apparatus according to claim 26, wherein the n-gram frequency element is further configured to calculate the n-gram frequency score only for ones of the plurality of languages for which the alphabet score meets the threshold.

28. An apparatus according to claim 26, wherein the n-gram frequency element is further configured to assign the entry a predetermined alphabet score in response to the alphabet score failing to meet the threshold.

29. An apparatus according to claim 21, further comprising a package ordering element configured to select the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.

30. An apparatus according to claim 21, wherein the entry is one of a plurality of segments comprising an input text and wherein the processing element is further configured to determine the language associated with the input text based on a cumulative combination of alphabet scores and n-gram frequency scores of each one of the plurality of segments.

31. An apparatus according to claim 21, wherein the apparatus is embodied as a mobile terminal.

32. An apparatus comprising:

means for receiving an entry in a computer readable text format;

means for determining an alphabet score of the entry for each of a plurality of languages;

means for determining an n-gram frequency score of the entry for each of the plurality of languages; and

means for determining a language associated with the entry based on a combination of the alphabet score and the n-gram frequency score.

33. An apparatus according to claim 32, further comprising means for selecting the language associated with the entry based on a predefined order of languages in response to the combination of the alphabet score and the n-gram frequency score providing an indeterminate result with respect to the language associated with the entry.