Computing numeric representations of words in a high-dimensional space
First Claim
1. A method performed by one or more computers, the method comprising:
- receiving a request for a numeric representation for a particular word in a high-dimensional space;
determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word; and
providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by;
obtaining a set of training data, wherein the set of training data comprises sequences of words;
training a plurality of classifiers and an embedding function on the set of training data, wherein the embedding function receives an input word and maps the input word to a numeric representation in the high-dimensional space in accordance with a set of embedding function parameters, wherein each of the classifiers corresponds to a respective position surrounding the input word in a sequence of words, and wherein each of the classifiers processes the numeric representation of the input word to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word will be found in the corresponding position relative to the input word, and wherein training the embedding function comprises determining trained values of the embedding function parameters;
processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary;
associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space; and
storing the data set on the one or more non-transitory computer storage media.
2 Assignments
0 Petitions
Accused Products
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for computing numeric representations of words. One of the methods includes obtaining a set of training data, wherein the set of training data comprises sequences of words; training a classifier and an embedding function on the set of training data, wherein training the embedding function comprises obtained trained values of the embedding function parameters; processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numerical representation of each word in the vocabulary in the high-dimensional space; and associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space.
15 Citations
18 Claims
-
1. A method performed by one or more computers, the method comprising:
-
receiving a request for a numeric representation for a particular word in a high-dimensional space; determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word; and providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by; obtaining a set of training data, wherein the set of training data comprises sequences of words; training a plurality of classifiers and an embedding function on the set of training data, wherein the embedding function receives an input word and maps the input word to a numeric representation in the high-dimensional space in accordance with a set of embedding function parameters, wherein each of the classifiers corresponds to a respective position surrounding the input word in a sequence of words, and wherein each of the classifiers processes the numeric representation of the input word to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word will be found in the corresponding position relative to the input word, and wherein training the embedding function comprises determining trained values of the embedding function parameters; processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary; associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space; and storing the data set on the one or more non-transitory computer storage media. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A method performed by one or more computers, the method comprising:
-
receiving a request for a numeric representation for a particular word in a high-dimensional space; determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word; and providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by; obtaining a set of training data, wherein the set of training data comprises sequences of words; training a classifier and an embedding function on the set of training data, wherein the embedding function receives a plurality of words surrounding an unknown word in a sequence of words and maps the plurality of words into a numeric representation in accordance with a set of embedding function parameters, wherein the classifier processes the numeric representation of the sequence of words to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores measure a predicted likelihood that the corresponding word is the unknown word, and wherein training the embedding function comprises determining trained values of the embedding function parameters; processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary in the high-dimensional space; associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space; and storing the data set on the one or more non-transitory computer storage media. - View Dependent Claims (10, 11, 12, 13, 14, 15, 16)
-
-
17. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
-
receiving a request for a numeric representation for a particular word in a high-dimensional space; determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word; and providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by; obtaining a set of training data, wherein the set of training data comprises sequences of words; training a plurality of classifiers and an embedding function on the set of training data, wherein the embedding function receives an input word and maps the input word to a numeric representation in the high-dimensional space in accordance with a set of embedding function parameters, wherein each of the classifiers corresponds to a respective position surrounding the input word in a sequence of words, and wherein each of the classifiers processes the numeric representation of the input word to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word will be found in the corresponding position relative to the input word, and wherein training the embedding function comprises determining trained values of the embedding function parameters; processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary; associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space; and storing the data set on the one or more non-transitory computer storage media.
-
-
18. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
-
receiving a request for a numeric representation for a particular word in a high-dimensional space; determining, by accessing a data set that is stored on one or more non-transitory computer readable media and that associates each word in a vocabulary of words with a respective numeric representation of the word in the high-dimensional space, the numeric representation of the particular word; and providing the numeric representation of the particular word in response to the request, wherein the data set has been generated by; obtaining a set of training data, wherein the set of training data comprises sequences of words; training a classifier and an embedding function on the set of training data, wherein the embedding function receives a plurality of words surrounding an unknown word in a sequence of words and maps the plurality of words into a numeric representation in accordance with a set of embedding function parameters, wherein the classifier processes the numeric representation of the sequence of words to generate a respective word score for each word in a pre-determined set of words, wherein each of the respective word scores measure a predicted likelihood that the corresponding word is the unknown word, and wherein training the embedding function comprises determining trained values of the embedding function parameters; processing each word in the vocabulary using the embedding function in accordance with the trained values of the embedding function parameters to generate a respective numeric representation of each word in the vocabulary in the high-dimensional space; associating each word in the vocabulary with the respective numeric representation of the word in the high-dimensional space; and storing the data set on the one or more non-transitory computer storage media.
-
Specification