Translating terms using numeric representations

• US 9,805,028 B1
• Filed: 09/17/2015
• Issued: 10/31/2017
• Est. Priority Date: 09/17/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sutskever, Ilya, Mikolov, Tomas, Dean, Jeffrey Adgate, Le, Quoc V.
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): Kovacek, David
• Application Number:
  US14/857,709
• Time in Patent Office: 775 Days
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/44 Statistical methods, e.g. probability models
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06F 40/51 Translation evaluation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for translating terms using numeric representations. One of the methods includes obtaining data that associates each term in a vocabulary of terms in a first language with a respective high-dimensional representation of the term; obtaining data that associates each term in a vocabulary of terms in a second language with a respective high-dimensional representation of the term; receiving a first language term; and determining a translation into the second language of the first language term from the high-dimensional representation of the first language term and the high-dimensional representations of terms in the vocabulary of terms in the second language.

Claims

1. A method comprising:

maintaining data that associates each term in a vocabulary of terms in a first language with a respective high-dimensional representation of the term, wherein the high-dimensional representation of the term is a numeric representation of the term in a high-dimensional space, and wherein positions of high-dimensional representations of terms from the vocabulary of terms in the first language in the high-dimensional space reflect syntactic similarities, semantic similarities, or both between the terms from the vocabulary of terms in the first language;

maintaining data that associates each term in a vocabulary of terms in a second language with a respective high-dimensional representation of the term, wherein the high-dimensional representation of the term is a numeric representation of the term in the high-dimensional space, and wherein positions of high-dimensional representations of terms from the vocabulary of terms in the second language in the high-dimensional space reflect syntactic similarities, semantic similarities, or both between the terms from the vocabulary of terms in the second language;

receiving a first language term, wherein the first language term is a term from the vocabulary of terms in the first language; and

determining a translation into the second language of the first language term from the high-dimensional representation of the first language term and the high-dimensional representations of terms in the vocabulary of terms in the second language, wherein determining the translation into the second language of the first language term comprises;

identifying a high-dimensional representation of the first language term;

applying a transformation to the high-dimensional representation of the first language term to generate a transformed representation, wherein applying the transformation to the high-dimensional representation of the first language term comprises applying the transformation in accordance with trained values of a set of parameters, the trained values of the set of parameters having been determined through applying a machine learning training procedure on training terms in the first language and a respective translation of each of the training terms into the second language;

selecting, from the high-dimensional representations of the terms in the vocabulary of terms in the second language, a closest high-dimensional representation to the transformed representation; and

selecting the term in the second language that is associated with the closest high-dimensional representation as the translation into the second language of the first language term.

2. The method of claim 1, wherein selecting the closest high-dimensional representation to the transformed representation comprises selecting a high-dimensional representation having a largest cosine similarity value with the transformed representation.

3. The method of claim 2, further comprising:

determining a confidence score for the translation of the first language term from the largest cosine similarity value; and

determining that the translation of the first language term into the second language is valid when the confidence score exceeds a threshold value.

4. The method of claim 1, wherein the transformed representation z satisfies z=Wx, wherein W is a matrix of the trained values of the set of parameters and wherein x is the high-dimensional representation of the first language term.

5. The method of claim 1, further comprising:

determining the trained values of the set of parameters from initial values of the set of parameters.

6. The method of claim 5, wherein determining the trained values comprises:

selecting the training terms from the vocabulary of terms in the first language;

obtaining the respective translation into the second language of each of the training terms from a translation service; and

performing multiple iterations of the machine learning training procedure to determine the trained values using the training terms and the translations of the training terms as training data.

7. The method of claim 6, wherein performing multiple iterations of the machine learning training procedure comprises performing multiple iterations of the machine learning training procedure to determine a matrix of values W that satisfies:

8. The method of claim 6, wherein the machine learning training procedure is stochastic gradient descent.

9. The method of claim 6, wherein selecting the plurality of training terms from the vocabulary of terms in the first language comprises selecting a pre-determined number of most frequently occurring terms from the vocabulary of terms in the first language.

10. The method of claim 1, further comprising:

determining a distance between the high-dimensional representation of the selected translation and a high-dimensional representation of an existing translation into the second language of the first language term; and

determining that the existing translation is likely correct or likely incorrect based on the distance.

11. The method of claim 10, wherein the distance is a cosine similarity and wherein determining that the existing translation is likely correct or likely incorrect comprises:

determining that the existing translation is likely correct when the cosine similarity exceeds a threshold; and

determining that the existing translation is likely incorrect when the cosine similarity does not exceed the threshold.

12. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

maintaining data that associates each term in a vocabulary of terms in a first language with a respective high-dimensional representation of the term, wherein the high-dimensional representation of the term is a numeric representation of the term in a high-dimensional space, and wherein positions of high-dimensional representations of terms from the vocabulary of terms in the first language in the high-dimensional space reflect syntactic similarities, semantic similarities, or both between the terms from the vocabulary of terms in the first language;

maintaining data that associates each term in a vocabulary of terms in a second language with a respective high-dimensional representation of the term, wherein the high-dimensional representation of the term is a numeric representation of the term in the high-dimensional space, and wherein positions of high-dimensional representations of terms from the vocabulary of terms in the second language in the high-dimensional space reflect syntactic similarities, semantic similarities, or both between the terms from the vocabulary of terms in the second language;

receiving a first language term, wherein the first language term is a term from the vocabulary of terms in the first language; and

determining a translation into the second language of the first language term from the high-dimensional representation of the first language term and the high-dimensional representations of terms in the vocabulary of terms in the second language, wherein determining the translation into the second language of the first language term comprises;

identifying a high-dimensional representation of the first language term;

applying a transformation to the high-dimensional representation of the first language term to generate a transformed representation, wherein applying the transformation to the high-dimensional representation of the first language term comprises applying the transformation in accordance with trained values of a set of parameters, the trained values of the set of parameters having been determined through applying a machine learning training procedure on training terms in the first language and a respective translation of each of the training terms into the second language;

selecting, from the high-dimensional representations of the terms in the vocabulary of terms in the second language, a closest high-dimensional representation to the transformed representation; and

selecting the term in the second language that is associated with the closest high-dimensional representation as the translation into the second language of the first language term.

13. The system of claim 12, wherein selecting the closest high-dimensional representation to the transformed representation comprises selecting a high-dimensional representation having a largest cosine similarity value with the transformed representation.

14. The system of claim 13, the operations further comprising:

determining a confidence score for the translation of the first language term from the largest cosine similarity value; and

determining that the translation of the first language term into the second language is valid when the confidence score exceeds a threshold value.

15. The system of claim 12, wherein the transformed representation z satisfies z=Wx, wherein W is a matrix of the trained values of the set of parameters and wherein x is the high-dimensional representation of the first language term.

16. A computer program product encoded on one or more non-transitory storage media, the computer program product comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

maintaining data that associates each term in a vocabulary of terms in a first language with a respective high-dimensional representation of the term, wherein the high-dimensional representation of the term is a numeric representation of the term in a high-dimensional space, and wherein positions of high-dimensional representations of terms from the vocabulary of terms in the first language in the high-dimensional space reflect syntactic similarities, semantic similarities, or both between the terms from the vocabulary of terms in the first language;

maintaining data that associates each term in a vocabulary of terms in a second language with a respective high-dimensional representation of the term, wherein the high-dimensional representation of the term is a numeric representation of the term in the high-dimensional space, and wherein positions of high-dimensional representations of terms from the vocabulary of terms in the second language in the high-dimensional space reflect syntactic similarities, semantic similarities, or both between the terms from the vocabulary of terms in the second language;

receiving a first language term, wherein the first language term is a term from the vocabulary of terms in the first language; and

determining a translation into the second language of the first language term from the high-dimensional representation of the first language term and the high-dimensional representations of terms in the vocabulary of terms in the second language, wherein determining the translation into the second language of the first language term comprises;

identifying a high-dimensional representation of the first language term;

applying a transformation to the high-dimensional representation of the first language term to generate a transformed representation, wherein applying the transformation to the high-dimensional representation of the first language term comprises applying the transformation in accordance with trained values of a set of parameters, the trained values of the set of parameters having been determined through applying a machine learning training procedure on training terms in the first language and a respective translation of each of the training terms into the second language;

selecting, from the high-dimensional representations of the terms in the vocabulary of terms in the second language, a closest high-dimensional representation to the transformed representation; and

selecting the term in the second language that is associated with the closest high-dimensional representation as the translation into the second language of the first language term.