SHORT TEXT LANGUAGE DETECTION USING GEOGRAPHIC INFORMATION

• US 20100114559A1
• Filed: 10/30/2008
• Published: 05/06/2010
• Est. Priority Date: 10/30/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Kim, Yookyung, Guo, Shuang, Hu, Xian Xiang, Li, Xin
• Granted Patent: US 8,548,797 B2
• Time in Patent Office: Days
• US Class Current: 704/8
• CPC Class Codes: G06F 40/263 Language identification

Abstract

A content-providing entity receives a relatively short text from a user and attempts to determine, automatically, based on that short text (and on other available clues), a language that the user can read and understand. The content-providing entity may then provide, to the user, documents that are written in the determined language. The content-providing entity may determine a language of the input text based on several factors in combination: (a) the service provider'"'"'s “market,” which is determined based on at least a portion of the URL of the Internet site to which the user directed his browser; (b) the user'"'"'s “region,” which is determined based on the source Internet Protocol (IP) address of the IP packets that the user sends to the Internet site; (c) the “script” in which the short user-entered text is written; and (d) a statistical analysis of the frequency of the characters present in the short user-entered text.

Claims

1. A computer-implemented method comprising:

determining a particular language based at least in part on both content of text submitted by a user and at least one of;

(a) whether the text is encoded in ASCII, (b) a top-level domain of a particular URL, (c) a source IP address that is associated with the user, and (d) whether characters of the text are in a specified subset of Unicode; and

presenting, to the user, one or more content items that are associated with the particular language.

wherein the content of the text does not expressly state the particular language.

2. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings; and

in response to determining that (a) the text is encoded in ASCII and (b) only one language is mapped to the particular market in the stored set of market-to-language mappings, determining that the particular language is the one language that is mapped to the particular market in the stored set of market-to-language mappings.

3. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, and (c) the top-level domain is within the specified subset of top-level domains, determining that the particular language is English.

4. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score satisfies the particular threshold, and (e) the characters of the text are all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

5. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the country in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language matches the language identified by the statistical textual analysis, determining that the particular language is the language identified by the statistical textual analysis;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

6. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the region in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language does not match the language identified by the statistical textual analysis, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

7. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, and (e) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

8. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is not encoded in ASCII and (b) the top-level domain is within the specified subset of top-level domains, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

9. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, and (c) the confidence score satisfies the particular threshold, determining that the particular language is a language identified by the statistical textual analysis.

10. The method of claim 1, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether characters of the text are all within a Latin subset of Unicode;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold; and

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, (c) the confidence score satisfies the particular threshold, and (d) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

11. A volatile or non-volatile computer-readable storage medium storing one or more instructions which, when executed by one or more processors, cause the one or more processors to perform steps comprising:

determining a particular language based at least in part on both content of text submitted by a user and at least one of;

(a) whether the text is encoded in ASCII, (b) a top-level domain of a particular URL, (c) a source IP address that is associated with the user, and (d) whether characters of the text are in a specified subset of Unicode; and

presenting, to the user, one or more content items that are associated with the particular language.

wherein the content of the text does not expressly state the particular language.

12. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings; and

in response to determining that (a) the text is encoded in ASCII and (b) only one language is mapped to the particular market in the stored set of market-to-language mappings, determining that the particular language is the one language that is mapped to the particular market in the stored set of market-to-language mappings.

13. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, and (c) the top-level domain is within the specified subset of top-level domains, determining that the particular language is English.

14. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score satisfies the particular threshold, and (e) the characters of the text are all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.

15. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the region in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language matches the language identified by the statistical textual analysis, determining that the particular language is the language identified by the statistical textual analysis;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

16. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

determining a region based on the source IP address;

determining a region default language that is mapped to the region in a stored set of region-to-language mappings;

determining whether the region default language matches a language identified by the statistical textual analysis; and

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, (e) the characters of the text are all within a Latin subset of Unicode, and (f) the region default language does not match the language identified by the statistical textual analysis, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings;

wherein the stored set of region-to-language mappings differs from the stored set of market-to-language mappings.

17. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether only one language is mapped to the particular market in a stored set of market-to-language mappings;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

determining whether characters of the text are all within a Latin subset of Unicode;

in response to determining that (a) the text is encoded in ASCII, (b) more than one language is mapped to the particular market in the stored set of market-to-language mappings, (c) the top-level domain is not within the specified subset of top-level domains, (d) the confidence score does not satisfy the particular threshold, and (e) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

18. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

in response to determining that (a) the text is not encoded in ASCII and (b) the top-level domain is within the specified subset of top-level domains, determining that the particular language is a market default language that is mapped to the particular market in the stored set of market-to-language mappings.

19. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold that is mapped to the particular market in a stored set of market-to-threshold mappings;

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, and (c) the confidence score satisfies the particular threshold, determining that the particular language is a language identified by the statistical textual analysis.

20. The computer-readable medium of claim 11, wherein determining the particular language comprises:

determining a particular market based on the top-level domain;

determining whether the text is encoded in ASCII;

determining whether the top-level domain is within a specified subset of top-level domains that consists only of “

cn,”

“

tw,”

“

jp,”

“

kr,” and

“

hk”

; and

determining whether characters of the text are all within a Latin subset of Unicode;

determining whether a confidence score, which reflects a level of confidence that a statistical textual analysis of the content of the text actually identified a language in which the content of the text is written, satisfies a particular threshold; and

in response to determining that (a) the text is not encoded in ASCII, (b) the top-level domain is not within the specified subset of top-level domains, (c) the confidence score satisfies the particular threshold, and (d) the characters of the text are not all within a Latin subset of Unicode, determining that the particular language is a language identified by the statistical textual analysis.