Methods and systems for selecting a language for text segmentation

• US 7,996,208 B2
• Filed: 09/30/2004
• Issued: 08/09/2011
• Est. Priority Date: 09/30/2004
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mandelson, Jacob Leon, Elbaz, Gilad Israel
• Primary Examiner(s): Jackson; Jakieda R
• Application Number:
  US10/955,660
• Publication Number:
  US 20060074628A1
• Time in Patent Office: 2,504 Days
• Field of Search: 704/8, 707/9
• US Class Current: 704/8
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

Methods and systems for selecting a language for text segmentation are disclosed. In one embodiment, at least a first candidate language and a second candidate language associated with a string of characters are identified, at least a first segmented result associated with the first candidate language and a second segmented result associated with the second candidate language are determined, a first frequency of occurrence for the first segmented result and a second frequency of occurrence for the second segmented result are determined, and an operable language is identified from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence.

Claims

1. A computer-implemented method, comprising:

receiving from a user of a computing device, at a computer server system, a request for information about one or more internet-accessible documents, the request having a string of characters;

identifying, using the computer server system, at least a first candidate language and a second candidate language associated with the request;

determining at least a first segmented result associated with the first candidate language from the string of characters and a second segmented result associated with the second candidate language from the string of characters;

determining a first frequency of occurrence for the first segmented result in a group of articles that are associated by the system with the first language and a second frequency of occurrence for the second segmented result in a group of articles that are associated by the system with the second language;

identifying, with the computer server system, an operable language from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence;

selecting, for use by the user of the computing device, electronic content in the identified operable language from among available content in multiple languages; and

providing the selected content to the computing device so that the selected content is arranged to be displayed to the user in the identified operable language and accompanying the requested one or more internet-accessible documents.

2. The method of claim 1, wherein identifying the operable language comprises identifying an operable segmented result from the first segmented result and the second segmented result based at least in part on the first frequency of occurrence and the second frequency of occurrence.

3. The method of claim 1, wherein:

the received string of characters is in an unknown language;

the first and second candidate languages are identified as candidates for the unknown language from among a plurality of languages based in part on at least one language signal determined from an electronic communication received from the user that submitted the string of characters; and

the identified candidates comprise fewer than all of the plurality of languages; and

wherein determining segmented results, determining frequencies of occurrence, and identifying the operable language are performed using the identified candidates for unknown language.

4. The method of claim 3, wherein the at least one language signal comprises at least one of linguistics associated with the string of characters, an IP address of the user, a character set associated with the string of characters, a browser setting of a browser application program associated with the user, and a top-level domain associated with the string of characters.

5. The method of claim 1, wherein the string of characters comprises a domain name submitted by the user.

6. The method of claim 1, wherein the first segmented result comprises a first combination of tokens and the second segmented result comprises a second combination of tokens, wherein the first combination of tokens and the second combination of tokens are generated by separating the string of characters at a position between two characters that are immediately adjacent each other in the string of characters.

7. The method of claim 1, wherein determining the first frequency of occurrence for the first segmented result comprises identifying a normalized value of a number of times the first segmented result appears in the first group of articles and identifying a normalized value of a number of times the second segmented result appears in the second group of articles.

8. The method of claim 7, further comprising determining a number of articles that contain the first segmented result by accessing an index of articles.

9. The method of claim 1, wherein determining the first frequency of occurrence for the first segmented result comprises determining a number of article identifiers in a search result set that has been generated in response to a search query comprising the first segmented result.

10. The method of claim 1, wherein determining the first frequency of occurrence for the first segmented result comprises determining a number of occurrences of the first segmented result in a plurality of search queries in the first candidate language and normalizing the number of occurrences based on a total number of search queries in the first candidate language.

11. The method of claim 1, wherein the string of characters comprises a domain name and determining a first segmented result and a second segmented result comprises parsing the domain name into a pair of tokens at a plurality of locations along the string of characters.

12. The method of claim 1, wherein providing the electronic content comprises identifying one or more keywords from the first segmented result or the second segmented result, and selecting electronic content that matches the identified one or more keywords.

13. The method of claim 12, wherein the electronic content comprises at least one of an advertisement, a web page, and a status message.

14. The method of claim 1, wherein determining the first segmented result comprises:

determining a plurality of segmented results in the first candidate language from the string of characters; and

identifying the first segmented result from the plurality of segmented results.

15. The method of claim 14, wherein identifying the first segmented result comprises calculating a probability value for each of the plurality of segmented results.

16. The method of claim 15, wherein a first probability value associated with the first segmented result is based at least in part on a frequency of each token within the first segmented result.

17. The method of claim 1, wherein identifying the operable language comprises identifying a number of hits of first segmented result against a database of content in the first candidate language and a number of hits of the second segmented result against a database of content in the second candidate language.

18. The computer-implemented method of claim 1, wherein each article in the group of articles that are associated by the system with the first language includes content in the first language;

wherein each article in the group of articles that are associated by the system with the second language includes content in the second language; and

wherein the first language is different than the second language.

19. A tangible and non-transitory computer-readable medium containing program code executable on a computer, comprising:

program code for receiving from a user of a computing device, at a computer server system, a request to receive one or more internet-accessible documents, the request having a string of characters;

program code for identifying at least a first candidate language and a second candidate language associated with a string of characters received in the request;

program code for determining at least a first segmented result associated with the first candidate language from the string of characters and a second segmented result associated with the second candidate language from the string of characters;

program code for determining a first frequency of occurrence for the first segmented result in a group of articles that are associated by the computer server system with the first language and a second frequency of occurrence for the second segmented result in a group of articles that are associated by the computer server system with the second language;

program code for identifying an operable language from the first candidate language and the second candidate language based at least in part on the first frequency of occurrence and the second frequency of occurrence;

program code for selecting, for use by the user of the computing device, electronic content in the identified operable language from among available content in multiple languages; and

program code for providing the selected content to the computing device so that the selected content is arranged to be displayed to the user in the identified operable language with the requested one or more internet-accessible documents.

20. The computer-readable medium of claim 19, wherein program code for identifying the operable language comprises program code for identifying an operable segmented result from the first segmented result and the second segmented result based at least in part on the first frequency of occurrence and the second frequency of occurrence.

21. The computer-readable medium of claim 19, wherein:

the received string of characters is in an unknown language;

the first and second candidate languages are identified as candidates for the unknown language from among a plurality of languages based in part on at least one language signal determined from an electronic communication received from the user that submitted the string of characters; and

the identified candidates comprise fewer than all of the plurality of languages; and

wherein determining segmented results, determining frequencies of occurrence, and identifying the operable language are performed using the identified candidates for unknown language.

22. The computer-readable medium of claim 21, wherein the at least one language signal comprise at least one of linguistics associated with the string of characters, an IP address of the user, a character set associated with the string of characters, a browser setting of a browser application program associated with the user, and a top-level domain associated with the string of characters.

23. The computer-readable medium of claim 19, wherein the string of characters comprises a domain name submitted by the user.

24. The computer-readable medium of claim 19, wherein the first segmented result comprises a first combination of tokens and the second segmented result comprises a second combination of tokens, wherein the first combination of tokens and the second combination of tokens are generated by separating the string of characters at a position between two characters that are immediately adjacent each other in the string of characters.

25. The computer-readable medium of claim 19, wherein program code for determining the first frequency of occurrence for the first segmented result comprises program code for identifying a normalized value of a number of times the first segmented result appears in the first group of articles and identifying a normalized value of a number of times the second segmented result appears in the second group of articles.

26. The computer-readable medium of claim 25, wherein program code for determining the first frequency of occurrence for the first segmented result comprises program code for determining a number of article identifiers in a search result set generated in response to a search query comprising the first segmented result.

27. The computer-readable medium of claim 26, further comprises program code for determining a number of articles containing the first segmented result by accessing an index of articles.

28. The computer-readable medium of claim 19, wherein program code for determining the first frequency of occurrence for the first segmented result comprises program code for determining a number of occurrences of the first segmented result in a plurality of search queries in the first candidate language and normalizing the number of occurrences based on a total number of search queries in the first candidate language.

29. The computer-readable medium of claim 19, wherein the string of characters comprises a domain name and determining a first segmented result and a second segmented result comprises parsing the domain name into a pair of tokens at a plurality of locations along the string of characters.

30. The computer-readable medium of claim 19, wherein providing the electronic content comprises identifying one or more keywords from the first segmented result or the second segmented result, and selecting electronic content that matches the identified one or more keywords.

31. The computer-readable medium of claim 30, wherein the electronic content comprises at least one of an advertisement, a web page, and a status message.

32. The computer-readable medium of claim 19, wherein program code for determining the first segmented result comprises:

program code for determining a plurality of segmented results from the string of characters; and

program code for identifying the first segmented result from the plurality of segmented results.

33. The computer-readable medium of claim 32, wherein program code for identifying the first segmented result comprises program code for calculating a probability value for each of the plurality of segmented results.

34. The computer-readable medium of claim 33, wherein a first probability value associated with the first segmented result is based at least in part on a frequency of each token within the first segmented result.

35. The computer-readable medium of claim 19, wherein program code for identifying the operable language comprises identifying a number of hits of first segmented result against a database of content in the first candidate language and a number of hits of the second segmented result against a database of content in the second candidate language.

36. A computer-implemented method, comprising:

receiving from a user of a computing device, at a computer server system, a request to receive one or more internet-accessible documents, the request having a string of characters that include a domain name;

determining at least a first segmented result in a first candidate language and at least a second segmented result in a second candidate language from the domain name;

determining at least a first frequency of occurrence for the first segmented result in a group of articles that are associated with the first language and based at least in part on at least one of an article index, a text index, and a search result set;

determining a second frequency of occurrence for the second segmented result in a group of articles that are associated with the second language and;

if the first frequency of occurrence is greater than the second frequency of occurrence, then selecting the first candidate language as an operable language;

if the second frequency of occurrence is greater than the first frequency of occurrence, then selecting the second candidate language as the operable language;

selecting an advertisement from among available advertisements in multiple languages based at least in part on the operable language, wherein the advertisement includes text in the operable language; and

providing the selected advertisement to the computing device arranged to be displayed to the user with the requested one or more internet-accessible documents associated with the domain name.