Methods and systems for selecting a language for text segmentation

• US 20060074628A1
• Filed: 09/30/2004
• Published: 04/06/2006
• Est. Priority Date: 09/30/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Elbaz, Gilad Israel, Mandelson, Jacob Leon
• Granted Patent: US 7,996,208 B2
• Time in Patent Office: Days
• US Class Current: 704/8
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

Methods and systems for selecting a language for text segmentation are disclosed. In one embodiment, at least a first candidate language and a second candidate language associated with a string of characters are identified, at least a first segmented result associated with the first candidate language and a second segmented result associated with the second candidate language are determined, a first frequency of occurrence for the first segmented result and a second frequency of occurrence for the second segmented result are determined, and an operable language is identified from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence.

Claims

1. A method, comprising:

identifying at least a first candidate language and a second candidate language associated with a string of characters;

determining at least a first segmented result associated with the first candidate language from the string of characters and a second segmented result associated with the second candidate language from the string of characters;

determining a first frequency of occurrence for the first segmented result and a second frequency of occurrence for the second segmented result; and

identifying an operable language from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence.

2. The method of claim 1, wherein identifying the operable language comprises identifying an operable segmented result from the first segmented result and the second segmented result based at least in part on the first frequency of occurrence and the second frequency of occurrence.

3. The method of claim 1, wherein identifying a first candidate language and a second candidate language is based in part on at least one language signal.

4. The method of claim 3, wherein the at least one language signal comprises at least one of linguistics associated with the string of characters, an IP address of a user associated with the string of characters, a character set associated with the string of characters, a browser setting of a browser application program associated with a user associated with the string of characters, and a top-level domain associated with the string of characters.

5. The method of claim 1, wherein identifying the operable language is based at least in part on at least one language signal.

6. The method of claim 1, wherein the first segmented result comprises a first combination of tokens and the second segmented result comprises a second combination of tokens.

7. The method of claim 1, wherein determining the first frequency of occurrence for the first segmented result comprises determining a number of articles in the first candidate language containing the first segmented result and normalizing the number of articles based on a total number of articles in the first candidate language.

8. The method of claim 7, wherein determining the number of articles in the first candidate language containing the first segmented result comprises determining a number of article identifiers in a search result set generated in response to a search query comprising the first segmented result.

9. The method of claim 7, wherein determining the number of articles containing the first segmented result comprises accessing an index of articles.

10. The method of claim 1, wherein determining the first frequency of occurrence for the first segmented result comprises determining a number of occurrences of the first segmented result in a plurality of search queries in the first candidate language and normalizing the number of occurrences based on a total number of search queries in the first candidate language.

11. The method of claim 1, wherein the string of characters comprises a domain name.

12. The method of claim 1, further comprising selecting an article based at least in part on the operable language.

13. The method of claim 12, wherein the article comprises at least one of an advertisement, a web page, and a status message.

14. The method of claim 1, wherein determining the first segmented result comprises:

determining a plurality of segmented results in the first candidate language from the string of characters; and

identifying the first segmented result from the plurality of segmented results.

15. The method of claim 14, wherein identifying the first segmented result comprises calculating a probability value for each of the plurality of segmented results.

16. The method of claim 15, wherein a first probability value associated with the first segmented result is based at least in part on a frequency of each token within the first segmented result.

17. The method of claim 1, wherein identifying the operable language from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence comprises selecting the first candidate language if the first frequency of occurrence is greater than the second frequency of occurrence.

18. A computer-readable medium containing program code, comprising:

program code for identifying at least a first candidate language and a second candidate language associated with a string of characters;

program code for determining at least a first segmented result associated with the first candidate language from the string of characters and a second segmented result associated with the second candidate language from the string of characters;

program code for determining a first frequency of occurrence for the first segmented result and a second frequency of occurrence for the second segmented result; and

program code for identifying an operable language from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence.

19. The computer-readable medium of claim 18, wherein identifying the operable language comprises program code for identifying an operable segmented result from the first segmented result and the second segmented result based at least in part on the first frequency of occurrence and the second frequency of occurrence.

20. The computer-readable medium of claim 18, wherein identifying a first candidate language and a second candidate language is based in part on at least one language signal.

21. The computer-readable medium of claim 20, wherein the at least one language signal comprise at least one of linguistics associated with the string of characters, an IP address of a user associated with the string of characters, a character set associated with the string of characters, a browser setting of a browser application program associated with a user associated with the string of characters, and a top-level domain associated with the string of characters.

22. The computer-readable medium of claim 18, wherein identifying the operable language is based at least in part on at least one language signal.

23. The computer-readable medium of claim 18, wherein the first segmented result comprises a first combination of tokens and the second segmented result comprises a second combination of tokens.

24. The computer-readable medium of claim 18, wherein program code for determining the first frequency of occurrence for the first segmented result comprises program code for determining a number of articles in the first candidate language containing the first segmented result and normalizing the number of articles based on a total number of articles in the first candidate language.

25. The computer-readable medium of claim 24, wherein program code for determining the number of articles in the first candidate language containing the first segmented result comprises program code for determining a number of article identifiers in a search result set generated in response to a search query comprising the first segmented result.

26. The computer-readable medium of claim 25, wherein program code for determining the number of articles containing the first segmented result comprises program code for accessing an index of articles.

27. The computer-readable medium of claim 18, wherein program code for determining the first frequency of occurrence for the first segmented result comprises program code for determining a number of occurrences of the first segmented result in a plurality of search queries in the first candidate language and normalizing the number of occurrences based on a total number of search queries in the first candidate language.

28. The computer-readable medium of claim 18, wherein the string of characters comprises a domain name.

29. The computer-readable medium of claim 18, further comprising program code for selecting an article based at least in part on the operable language.

30. The computer-readable medium of claim 29, wherein the article comprises at least one of an advertisement, a web page, and a status message.

31. The computer-readable medium of claim 18, wherein program code for determining the first segmented result comprises:

program code for determining a plurality of segmented results from the string of characters; and

program code for identifying the first segmented result from the plurality of segmented results.

32. The computer-readable medium of claim 31, wherein program code for identifying the first segmented result comprises program code for calculating a probability value for each of the plurality of segmented results.

33. The computer-readable medium of claim 32, wherein a first probability value associated with the first segmented result is based at least in part on a frequency of each token within the first segmented result.

34. The computer-readable medium of claim 18, wherein program code for identifying the operable language from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence comprises program code for selecting the first candidate language if the first frequency of occurrence is greater than the second frequency of occurrence.

35. A method, comprising:

determining at least a first segmented result in a first candidate language and at least a second segmented result in a second candidate language from a domain name;

determining at least a first frequency of occurrence for the first segmented result based at least in part on at least one of an article index, a text index, and a search result set;

determining a second frequency of occurrence for the second segmented result;

if the first frequency of occurrence is greater than the second frequency of occurrence, then selecting the first candidate language as an operable language;

if the second frequency of occurrence is greater than the first frequency of occurrence, then selecting the second candidate language as the operable language;

selecting an advertisement based at least in part on the operable language, wherein the advertisement includes text in the operable language; and

causing a display of the advertisement in association with a web page associated with the domain name.