Exponential Modeling with Deep Learning Features

• US 20200134466A1
• Filed: 10/16/2019
• Published: 04/30/2020
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Weintraub, Mitchel, Suresh, Ananda Theertha, Variani, Ehsan
• Granted Patent: US 11,568,260 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/063 using electronic means
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent

Abstract

Aspects of the present disclosure enable humanly-specified relationships to contribute to a mapping that enables compression of the output structure of a machine-learned model. An exponential model such as a maximum entropy model can leverage a machine-learned embedding and the mapping to produce a classification output. In such fashion, the feature discovery capabilities of machine-learned models (e.g., deep networks) can be synergistically combined with relationships developed based on human understanding of the structural nature of the problem to be solved, thereby enabling compression of model output structures without significant loss of accuracy. These compressed models provide improved applicability to “on device” or other resource-constrained scenarios.

Claims

1. A computer system, comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store a machine-learned classification model configured to generate a classification output that comprises a plurality of classification scores respectively for a number of discrete classes based on a set of input data, the classification score for each discrete class indicative of a likelihood that the input data corresponds to the discrete class;

wherein the machine-learned classification model comprises an embedding model and an exponential model;

wherein the embedding model is configured to receive the set of input data and produce an embedding based on the set of input data, wherein the embedding comprises a number of parameter values respectively for a number of parameters included in a final layer of the embedding model, and wherein the number of parameter values is less than the number of discrete classes; and

wherein the exponential model is configured to receive the embedding and apply a mapping to generate the classification output, wherein the mapping describes a plurality of relationships between the number of parameters included in the final layer of the embedding model and the number of discrete classes.

2. The computer system of claim 1, wherein the exponential model comprises a maximum entropy model.

3. The computer system of claim 1, wherein the plurality of relationships specify, for each of the number of discrete classes, one or more of the number of parameters to which such discrete class is mapped.

4. The computer system of claim 1, wherein, for each of the number of discrete classes, the exponential model determines its respective classification score based at least in part on a first exponential raised to the power of a first sum of the parameter values to which such discrete class is mapped by the mapping divided by a sum of all possible output values.

5. The computer system of claim 1, wherein at least one of the discrete classes is mapped to two or more of the parameters included in the final layer of the embedding model.

6. The computer system of claim 1, wherein at least one of the parameters included in the final layer of the embedding model is mapped to two or more of the discrete classes.

7. The computer system of claim 1, wherein at least one of the plurality of relationships described by the mapping is user-specified.

8. The computer system of claim 1, wherein at least one of the plurality of relationships described by the mapping is deterministically assigned.

9. The computer system of claim 1, wherein at least one of the plurality of relationships is machine-learned.

10. The computer system of claim 1, wherein the number of discrete classes comprise a number of discrete words included in a vocabulary.

11. The computer system of claim 10, wherein at least one of the plurality of relationships described by the mapping is based on the presence of one or more specified characters within each discrete word.

12. The computer system of claim 10, wherein at least one of the plurality of relationships described by the mapping is based on the presence of a specified string of two or more characters within each discrete word.

13. The computer system of claim 1, wherein the number of discrete classes comprise a number of discrete items available for selection by a recommender system.

14. The computer system of claim 1, wherein at least one of the plurality of relationships described by the mapping is based on a shared characteristic among a subset of the plurality of discrete classes.

15. The computer system of claim 1, wherein the embedding model comprises one or more additional layers positioned prior to the final layer of the embedding model such that the final layer of the embedding model does not directly receive the set of input data.

16. The computer system of claim 1 wherein the final layer of the embedding model directly receives the set of input data such that the final layer is the only layer of the embedding model.

17. The computer system of claim 1 wherein the number of parameters of the last layer of the embedding model is a user-specified hyperparameter.

18. The computer system of claim 1 wherein the machine-learned classification model has been trained on a set of labeled training data using a supervised learning technique, wherein the supervised learning technique includes backpropagating a gradient of a loss function through the plurality of parameters according to the mapping.

19. A computer-implemented method, comprising:

obtaining, by one or more computing devices, a set of input data;

inputting, by the one or more computing devices, the set of input data into a machine-learned classification model configured to generate a classification output that comprises a plurality of classification scores respectively for a number of discrete classes based on a set of input data, the classification score for each discrete class indicative of a likelihood that the input data corresponds to the discrete class, wherein the machine-learned classification model comprises an embedding model and an exponential model, wherein the embedding model is configured to receive the set of input data and produce an embedding based on the set of input data, wherein the embedding comprises a number of parameter values respectively for a number of parameters included in a final layer of the embedding model, and wherein the number of parameter values is less than the number of discrete classes, and wherein the exponential model is configured to receive the embedding and apply a mapping to generate the classification output, wherein the mapping describes a plurality of relationships between the number of parameters included in the final layer of the embedding model and the number of discrete classes; and

receiving, by the one or more computing devices, a classification output of the machine-learned classification model.

20. The computer-implemented method of claim 19, further comprising, training the machine-learned classification model on a set of labeled training data using a supervised learning technique.