Short text language detection using geographic information

• US 8,548,797 B2
• Filed: 10/30/2008
• Issued: 10/01/2013
• Est. Priority Date: 10/30/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Kim, Yookyung, Guo, Shuang, Hu, Xian Xiang, Li, Xin
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US12/262,145
• Publication Number:
  US 20100114559A1
• Time in Patent Office: 1,797 Days
• Field of Search: 704/1, 704/3, 704/4, 704/8, 704/9, 704/270, 704/270.1, 704/275, 704/10
• US Class Current: 704/8
• CPC Class Codes: G06F 40/263 Language identification

Abstract

A content-providing entity receives a relatively short text from a user and attempts to determine, automatically, based on that short text (and on other available clues), a language that the user can read and understand. The content-providing entity may then provide, to the user, documents that are written in the determined language. The content-providing entity may determine a language of the input text based on several factors in combination: (a) the service provider'"'"'s “market,” which is determined based on at least a portion of the URL of the Internet site to which the user directed his browser; (b) the user'"'"'s “region,” which is determined based on the source Internet Protocol (IP) address of the IP packets that the user sends to the Internet site; (c) the “script” in which the short user-entered text is written; and (d) a statistical analysis of the frequency of the characters present in the short user-entered text.

Claims

1. A computer-implemented method comprising:

determining a particular language based at least in part on both content of text submitted by a user and a source IP address that is associated with the user; and

based on having determined the particular language for the user, presenting, to the user, one or more content items that are associated with the particular language;

wherein the content of the text does not expressly state the particular language.

2. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings; and

in response to determining that (a) the text is encoded in ASCII and (b) only one language is mapped to the particular market in the stored set of market-to-language mappings, determining that the particular language is the one language that is mapped to the particular market in the stored set of market-to-language mappings.

3. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, and (c) the top-level domain is within the specified subset of top-level domains, determining that the particular language is English.

4. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score satisfies the particular threshold, and (e) the characters of the text are all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

5. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the country in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language matches the language identified by the statistical textual analysis, determining that the particular language is the language identified by the statistical textual analysis;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

6. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the region in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language does not match the language identified by the statistical textual analysis, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

7. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, and (e) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

8. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is not encoded in ASCII and (b) the top-level domain is within the specified subset of top-level domains, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

9. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, and (c) the confidence score satisfies the particular threshold, determining that the particular language is a language identified by the statistical textual analysis.

10. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether characters of the text are all within a Latin subset of Unicode;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold; and

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, (c) the confidence score satisfies the particular threshold, and (d) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

11. The computer-readable medium of claim 10, wherein the context of text includes search query terms submitted by the user to a search engine to search for documents related to the search query terms.

12. The method of claim 1, wherein the context of text includes search query terms submitted by the user to a search engine to search for documents related to the search query terms.

13. A volatile or non-volatile computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the one or more processors to perform steps comprising:

determining a particular language based at least in part on both content of text submitted by a user and a source IP address that is associated with the user; and

based on having determined the particular language for the user, presenting, to the user, one or more content items that are associated with the particular language;

wherein the content of the text does not expressly state the particular language.

14. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings; and

in response to determining that (a) the text is encoded in ASCII and (b) only one language is mapped to the particular market in the stored set of market-to-language mappings, determining that the particular language is the one language that is mapped to the particular market in the stored set of market-to-language mappings.

15. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, and (c) the top-level domain is within the specified subset of top-level domains, determining that the particular language is English.

16. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score satisfies the particular threshold, and (e) the characters of the text are all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

17. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the region in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language matches the language identified by the statistical textual analysis, determining that the particular language is the language identified by the statistical textual analysis;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

18. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the region in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language does not match the language identified by the statistical textual analysis, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

19. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, and (e) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

20. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is not encoded in ASCII and (b) the top-level domain is within the specified subset of top-level domains, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

21. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, and (c) the confidence score satisfies the particular threshold, determining that the particular language is a language identified by the statistical textual analysis.

22. The computer-readable medium of claim 13, wherein determining the particular language comprises:

determining a particular market based on a top-level domain of a particular URL;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether characters of the text are all within a Latin subset of Unicode;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold; and

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, (c) the confidence score satisfies the particular threshold, and (d) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

23. A computer-implemented method comprising:

receiving one or more query terms of a query to a search engine that is capable of performing searches based on any one of a plurality of languages;

selecting a particular language, from among the plurality of languages, based at least in part on both the one or more query terms of the query and at least one of;

(a) whether the one or more query terms are encoded in ASCII, (b) a top-level domain of a particular URL, (c) a source IP address that is associated with a user entering the query, and (d) whether characters of the one or more query terms are in a specified subset of Unicode;

based on having selected the particular language, performing a search for content items associated with the particular language based on the one or more query terms; and

presenting, to the user, one or more content items that are associated with the particular language;

wherein the content of the one or more query terms does not expressly state the particular language.

24. The method of claim 23, wherein the particular language is determined based on whether the one or more query terms are encoded in ASCII.

25. The method of claim 23, wherein the particular language is determined based on a top-level domain of a particular URL.

26. The method of claim 23, wherein the particular language is determined based on a source IP address that is associated with a user entering the query.

27. The method of claim 23, wherein the particular language is determined based on whether characters of the one or more query terms are in a specified subset of Unicode.

28. A volatile or non-volatile computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the one or more processors to perform steps comprising:

receiving one or more query terms of a query to a search engine that is capable of performing searches based on any one of a plurality of languages;

selecting a particular language, from among the plurality of languages, based at least in part on both the one or more query terms of the query and at least one of;

(a) whether the one or more query terms are encoded in ASCII, (b) a top-level domain of a particular URL, (c) a source IP address that is associated with a user entering the query, and (d) whether characters of the one or more query terms are in a specified subset of Unicode;

based on having selected the particular language, performing a search for content items associated with the particular language based on the one or more query terms; and

presenting, to the user, one or more content items that are associated with the particular language;

wherein the content of the one or more query terms does not expressly state the particular language.

29. The computer-readable medium of claim 28, wherein the particular language is determined based on whether the one or more query terms are encoded in ASCII.

30. The computer-readable medium of claim 28, wherein the particular language is determined based on a top-level domain of a particular URL.

31. The computer-readable medium of claim 28, wherein the particular language is determined based on a source IP address that is associated with a user entering the query.

32. The computer-readable medium of claim 28, wherein the particular language is determined based on whether characters of the one or more query terms are in a specified subset of Unicode.