Systems and methods for language detection
First Claim
1. A computer-implemented method of identifying a language of a message, the method comprising:
- training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages;
performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages;
providing one or more combinations of the score sets as input to one or more distinct classifiers including the first classifier;
obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and
identifying the language of the message based on one of the confidence scores.
6 Assignments
0 Petitions
Accused Products
Abstract
Implementations of the present disclosure are directed to a method, a system, and a computer program storage device for detecting a language in a text message. A plurality of different language detection tests are performed on a message associated with a user. Each language detection test determines a set of scores representing a likelihood that the message is in one of a plurality of different languages. One or more combinations of the score sets are provided as input to one or more distinct classifiers. Output from each of the classifiers includes a respective indication that the message is in one of the different languages. The language in the message may be identified as being the indicated language from one of the classifiers, based on a confidence score and/or an identified linguistic domain.
216 Citations
30 Claims
-
1. A computer-implemented method of identifying a language of a message, the method comprising:
-
training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages; performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages; providing one or more combinations of the score sets as input to one or more distinct classifiers including the first classifier; obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and identifying the language of the message based on one of the confidence scores. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10)
-
-
11. A system for identifying a language of a message, the system comprising:
-
a computer storage device having instructions stored thereon; and a data processing apparatus configured to execute the instructions to perform operations comprising; training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages; performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages; providing one or more combinations of the score sets as input to one or more distinct classifiers including the classifier; obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and identifying the language of the message based on one of the confidence scores. - View Dependent Claims (12, 13, 14, 15, 16, 17, 18, 19, 20)
-
-
21. A computer program product stored in one or more storage devices for controlling a processing mode of a data processing apparatus, the computer program product being executable by the data processing apparatus to cause the data processing apparatus to perform operations comprising:
-
training a first classifier using training data comprising collections of first score sets from different language detection tests and an indication of the correct language for each collection of score sets wherein each first score set comprises a plurality of respective scores each representing a likelihood that a respective first message is in one of a plurality of different languages; performing a plurality of the language detection tests on text in a message authored by a user, each language detection test determining a respective set of scores, each score in the set of scores representing a likelihood that the message is in a respective language of the plurality of different languages; providing one or more combinations of the score sets as input to one or more distinct classifiers including the first classifier; obtaining as output from each of the one or more classifiers a respective indication that the message is in one of the plurality of different languages, the indication comprising a confidence score; and identifying the language of the message based on one of the confidence scores. - View Dependent Claims (22, 23, 24, 25, 26, 27, 28, 29, 30)
-
Specification