Language identification

• US 9,910,847 B2
• Filed: 09/30/2014
• Issued: 03/06/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Bittmann, Mark
• Primary Examiner(s): HARRIS, KEARA S
• Application Number:
  US14/502,465
• Publication Number:
  US 20160092427A1
• Time in Patent Office: 1,253 Days
• Field of Search: None
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A plurality of documents in each of a plurality of languages can be received. A Latent Semantic Indexing (LSI) index can be created from the plurality of documents. A language classification model can be trained from the LSI index. A document to be identified by language can be received. A vector in the LSI index can be generated for the document to be identified by language. The vector can be evaluated against the language classification model.

Claims

1. A computer-implemented method for language identification, comprising:

receiving, by at least one hardware processor, a plurality of documents in each of a plurality of languages;

creating, by the at least one hardware processor, a Latent Semantic Indexing (LSI) index from the plurality of documents;

training, by the at least one hardware processor, a language classification model based on the LSI index, where training the language classification model based on the LSI index comprises;

determining a vector for each of the plurality of documents in each of the plurality of languages from the LSI index, where the vectors are generated from singular value decomposition of the plurality of documents in each of a plurality of languages;

determining a combination of dimensions in the vector for each document that is indicative of the language of the document;

determining rules from the combinations of dimensions determined from the vectors for the plurality of documents; and

generating the language classification model from the rules;

receiving, by the at least one hardware processor, a document to be identified by language;

generating, by the at least one hardware processor, a vector in the LSI index for the document to be identified by language;

evaluating, by the at least one hardware processor, the vector of the document to be identified by language against the language classification model; and

identifying a language of the received document as being one of the plurality of languages based on the evaluating.

2. The method of claim 1, wherein the plurality of documents comprises a uniform number of documents in each language of the plurality of languages.

3. The method of claim 1, wherein the LSI index is a term vector index.

4. The method of claim 1, wherein training a language classification model from the LSI index comprises training the language classification model using one of:

a naï

ve Bayes classifier, a Bayesian network, a nearest neighbor search, and a C4.5 Decision Tree.

5. The method of claim 1, wherein

creating a Latent Semantic Indexing (LSI) index from the plurality of documents comprises:

partitioning, by the at least one hardware processor, the plurality of documents into a training subset and test subset, and creating the LSI index; and

creating, by the at least one hardware processor, the LSI index from the plurality of training documents; and

training, by the at least one hardware processor, a language classification model from the LSI index comprises;

training, by the at least one hardware processor, a language classification model using each of a plurality of machine learning techniques on the training subset;

measuring, by the at least one hardware processor, the performance of each model on the test subset;

selecting one of the models according to the measured performances; and

modeling, by the at least one hardware processor, language identification using the selected model.

6. The method of claim 5, wherein the performance of each model is measured using at least one of:

a true positive rate, a false positive rate, a precision, a recall, and an F-measure.

7. A computer program product, comprising:

a non-transitory computer-readable storage device having computer-executable program instructions embodied thereon that when executed by a computer cause the computer to identify a language of a communication, the computer-executable program instructions comprising;

computer-executable program instructions to receive a plurality of documents in each of a plurality of languages;

computer-executable program instructions to create a Latent Semantic Indexing (LSI) index from the plurality of documents;

computer-executable program instructions to train a language classification model based on the LSI index, wherein to train the language classification model based on the LSI index comprises;

determine a vector for each of the plurality of documents in each of the plurality of languages from the LSI index, where the vectors are generated from singular value decomposition of the plurality of documents in each of a plurality of languages;

for each document, determine a combination of dimensions in the vector for the document that is indicative of the language of the document;

determine rules from the combination of dimensions for each document for identifying a language of a document; and

generate the language classification model from the rules;

computer-executable program instructions to receive a document to be identified by language;

computer-executable program instructions to generate a vector in the LSI index for the document to be identified by language;

computer-executable program instructions to evaluate the vector of the document to be identified by language against the language classification model; and

computer-executable program instructions to identify a language of the received document as being in one of the plurality of languages based on an evaluation of the vector against the language classification model.

8. The computer program product of claim 7, wherein the plurality of documents comprises a uniform number of documents in each language of the plurality of languages.

9. The computer program product of claim 7, wherein the LSI index is a term vector index.

10. The computer program product of claim 7, wherein computer-executable program instructions to train a language classification model from the LSI index comprises computer-executable program instructions to train the language classification model using one of:

a naï

ve Bayes classifier, a Bayesian network, a nearest neighbor search, and a C4.5 Decision Tree.

11. The computer program product of claim 7, wherein

computer-executable program instructions to create a Latent Semantic Indexing (LSI) index from the plurality of documents comprise:

computer-executable program instructions to partition the plurality of documents into a training subset and test subset, and creating the LSI index; and

computer-executable program instructions to create the LSI index from the plurality of training documents; and

computer-executable program instructions to train a language classification model from the LSI index comprise;

computer-executable program instructions to train a language classification model using each of a plurality of machine learning techniques on the training subset;

computer-executable program instructions to measure the performance of each model on the test subset;

computer-executable program instructions to select one of the models according to the measured performances; and

computer-executable program instructions to model language identification using the selected model.

12. The computer program product of claim 11, wherein the performance of each model is measured using at least one of:

a true positive rate, a false positive rate, a precision, a recall, and an F-measure.

13. A system to identify the language of a communication, the system comprising:

a storage device; and

a processor communicatively coupled to the storage device, wherein the processor executes application code instructions received from the storage device to;

receive a plurality of documents in each of a plurality of languages;

create a Latent Semantic Indexing (LSI) index from the plurality of documents;

train a language classification model based on the LSI index, wherein to train the language classification model based on the LSI index comprises;

determine a vector for each of the plurality of documents in each of the plurality of languages from the LSI index, where the vectors are generated from singular value decomposition of the plurality of documents in each of a plurality of languages;

for each document, determine a combination of dimensions in the vector for the document that is indicative of the language of the document;

determine rules from the combination of dimensions for each document for identifying a language of a document; and

generate the language classification model from the rules;

receive a document to be identified by language;

generate a vector in the LSI index for the document to be identified by language;

evaluate the vector of the document to be identified by language against the language classification model; and

identify the language of the received document as being in one of the plurality of languages based on an evaluation of the vector against the language classification model.

14. The system of claim 13, wherein the plurality of documents comprises a uniform number of documents in each language of the plurality of languages.

15. The system of claim 13, wherein the LSI index is a term vector index.

16. The system of claim 13, wherein training a language classification model from the LSI index comprises training the language classification model using one of:

a naï

ve Bayes classifier, a Bayesian network, a nearest neighbor search, and a C4.5 Decision Tree.

17. The system of claim 13, wherein

creating a Latent Semantic Indexing (LSI) index from the plurality of documents comprises:

partitioning the plurality of documents into a training subset and test subset, and creating the LSI index; and

training a language classification model from the LSI index comprises;

training a language classification model using each of a plurality of machine learning techniques on the training subset;

measuring the performance of each model on the test subset;

selecting one of the models according to the measured performances; and

modeling language identification using the selected model.

18. The system of claim 17, wherein the performance of each model is measured using at least one of:

a true positive rate, a false positive rate, a precision, a recall, and an F-measure.