High-capacity machine learning system

• US 10,229,357 B2
• Filed: 09/11/2015
• Issued: 03/12/2019
• Est. Priority Date: 09/11/2015
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Jin, Ou, Bowers, Stuart Michael, Dzhulgakov, Dmytro
• Primary Examiner(s): Chen, Alan
• Application Number:
  US14/851,336
• Publication Number:
  US 20170076198A1
• Time in Patent Office: 1,278 Days
• Field of Search: None
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

The present disclosure is directed to a high-capacity training and prediction machine learning platform that can support high-capacity parameter models (e.g., with 10 billion weights). The platform implements a generic feature transformation layer for joint updating and a distributed training framework utilizing shard servers to increase training speed for the high-capacity model size. The models generated by the platform can be utilized in conjunction with existing dense baseline models to predict compatibilities between different groupings of objects (e.g., a group of two objects, three objects, etc.).

Claims

1. A method, comprising:

applying at least two different learning models to multiple data sets indicative of features associated with tuples of objects, each tuple being a pairing of a first object and a second object;

updating parameters of a prediction model based on a linear combination of at least two result sets corresponding to the at least two learning models;

receiving a request for identifying a compatible object for a given object based on respective features of the compatible object and the given object;

responsive to the received request, determining, based on the updated parameters of the prediction model, a prediction value associated with the given object, the prediction value indicative of a probability of compatibility based on the respective features; and

identifying the compatible object for the given object based on the prediction value.

2. The method of claim 1, further comprising:

organizing database shards of a database according to at least a first tiered set and a second tiered set associated with the at least two learning models, each tiered set having multiple tiers of shard servers, the multiple tiers of shard servers configured to perform a set of operations associated with a particular learning model of the at least two learning models and to store a respective result set associated with the particular learning model in the database shards.

3. The method of claim 2, wherein the multiple tiers of shard servers in the first tiered set includes a first tier of shard servers, a second tier of shard servers, and a third tier of shard servers.

4. The method of claim 3,

wherein each shard server of the first tier of shard servers is configured to perform a first operation associated with a first learning model of the at least two learning models;

wherein each shard server of the second tier of shard servers is configured to perform a second operation associated with the first learning model of the at least two learning models; and

wherein each shard server of the third tier of shard servers is configured to perform a third operation associated with the first learning model of the at least two learning models.

5. The method of claim 4, wherein the first operation associated with the first learning model is a pre-processing of a portion of the multiple data sets based on the first learning model.

6. The method of claim 4, wherein the second operation associated with the first learning model is an application of an algorithm associated with the first learning model to a portion of the multiple data sets.

7. The method of claim 4, wherein the third operation associated with the first learning model is providing a subset of a first result set resulting from an application of an algorithm associated with the first learning model to a portion of the multiple data sets.

8. The method of claim 7, further comprising:

aggregating subsets of the first result set received from individual shard servers of the third tier of shard servers; and

providing the aggregated subsets of the first result set for further processing.

9. The method of claim 2,

wherein the multiple tiers of shard servers in the second tiered set includes a first tier of shard servers, a second tier of shard servers, and a third tier of shard servers;

wherein each shard server of the first tier of shard servers is configured to perform a first operation associated with a second learning model of the at least two learning models;

wherein each shard server of the second tier of shard servers is configured to perform a second operation associated with the second learning model of the at least two learning models; and

wherein each shard server of the third tier of shard servers is configured to perform a third operation associated with the second learning model of the at least two learning models.

10. The method of claim 9,

wherein the first operation is a pre-processing of a portion of the multiple data sets;

wherein the second operation is an application of an algorithm associated with the second learning model to a portion of the multiple data sets;

wherein the third operation is providing a subset of a second result set resulting from an application of an algorithm associated with the second learning model to a portion of the multiple data sets.

11. The method of claim 10, further comprising:

aggregating subsets of the second result set received from individual shard servers of the third tier of shard servers; and

providing the aggregated subsets of the second result set for further processing.

12. The method of claim 11, further comprising:

updating a subset of the parameters of the prediction model based on the aggregated subsets of the second result set.

13. The method of claim 2, further comprising:

receiving a request for a training of the prediction model, the request including the multiple data sets;

translating the request for the training into sharded requests and providing each sharded request to a respective shard server associated with the particular learning model of the at least two learning models.

14. The method of claim 1, wherein the at least two learning models includes a first learning model that is a gradient boosted decision tree (GBDT) model.

15. The method of claim 1, wherein the at least two learning models includes a second learning model that is a neural network model.

16. A system, comprising:

a training system configured to;

preprocess multiple data sets associated with tuples of objects; and

analyze the multiple data sets to update parameters of a prediction model, wherein to analyze includes;

apply a first learning model to generate a first result set; and

apply a second learning model to generate a second result set,

wherein the first and second learning models are different learning models, and

wherein the parameters are updated based on a linear combination of the first and second result sets; and

a prediction system configured to;

determine, responsive to a request for identifying a compatible object for a given object, a prediction value associated with the given object based on the updated parameters of the prediction model; and

identify the compatible object for the given object based on the prediction value.

17. The system of claim 16, further comprising:

a database having database shards organized according to at least a first tiered set and a second tiered set associated with the first learning model and the second learning model respectively, each tiered set having multiple tiers of shard servers, the multiple tiers of shard servers configured to perform a set of operations associated with a particular learning model of the at least two learning models and to store a respective result set associated with the particular learning model in the database shards.

18. The system of claim 17,

wherein the multiple tiers of shard servers in the first tiered set includes a given tier of shard servers configured to apply an algorithm associated with the first learning model to a portion of the multiple data sets;

and

wherein the training system is further configured to aggregate subsets of the first result set received from individual shard servers of the given tier of shard servers, wherein a subset of the parameters is updated based on the aggregated subsets of the first result set.

19. A computer readable storage medium storing instructions that when executed by a processor causes the processor to implement a process, the instructions comprising:

instructions for preprocessing multiple data sets by parsing at least a first dataset and a second data set, the first data set including data indicative of a first feature shared between a first object and a second object, the second data set including a second feature shared between the first object and the second object;

instructions for applying a first learning model to the multiple data sets to generate a first result set;

instructions for applying a second learning model to the multiple data sets to generate a second result set, the first and second learning models being different learning models;

instructions for updating parameters of a prediction model based on a linear combination of the first result set and the second result set;

instructions for determining, in response to a request for identifying a compatible object for a given object, a prediction value associated with the given object based on the updated parameters of the prediction model, the prediction value indicative of a probability of compatibility; and

instructions for identifying the compatible object for the given object based on the prediction value.

20. The computer readable storage medium of claim 19, further comprising:

instructions for organizing database shards of a database according to at least a first tiered set and a second tiered set associated with the first learning model and the second learning model respectively, each tiered set having multiple tiers of shard servers, the multiple tiers of shard servers configured to perform a set of operations associated with a particular learning model of the at least two learning models and to store a respective result set associated with the particular learning model in the database shards;

instructions for configuring a given tier of shard servers in the first tiered set to apply an algorithm associated with the first learning model to a portion of the multiple data sets;

instructions for configuring a given tier of shard servers in the second tiered set to apply an algorithm associated with the second learning model to the portion of the multiple data sets;

instructions for aggregating subsets of the first result set received from individual shard servers of the given tier of shard servers in the first tiered set;

instructions for aggregating subsets of the second result set received from individual shard servers of the given tier of shard servers in the second tiered set;

instructions for performing linear combination on the aggregated subsets of the first result set and the aggregated subsets of the second result set to update the parameters of the prediction model.