Systems and methods for privacy-preserving generation of models for estimating consumer behavior

• US 10,430,727 B1
• Filed: 04/03/2019
• Issued: 10/01/2019
• Est. Priority Date: 04/03/2019
• Status: Active Grant

• Current Assignee: NFL Enterprises LLC (National Football League, Inc.)
• Original Assignee: NFL Enterprises LLC (National Football League, Inc.)
• Inventors: Fusillo, Iwao, Friedman, Justin, Yang, Robert Joseph, Highhill, David Samuel, Berman, Adriane Beth, Basandra, Nitish
• Primary Examiner(s): Hill, Stanley K.
• Application Number:
  US16/374,594
• Time in Patent Office: 181 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute

Abstract

A system, method, and computer readable storage media for training a model to estimate an unknown consumer behavior while preserving consumer privacy by combining a first training data set of a first organization with a second training data set of a second organization on a third-party computer system, wherein the second training data set contains an attribute value the first organization wishes to estimate, and wherein the first organization cannot access the third-party computing system, providing a trainable model definition to the third-party computer, training the model, and returning the trained computer model.

Claims

1. A computer-implemented method for training a model to estimate an unknown consumer behavior, comprising the steps of:

calculating, by a first computing system, a vector representing a consumer'"'"'s behavior by applying one or more vectorization rules to a set of behavioral attribute values for the consumer in a data set of a first organization;

transforming the vector, by the first computing system, into an estimated unknown consumer behavioral attribute value by applying a trained model, wherein the model was trained by;

providing, to a second computing system separate from the first computing system;

a first training data set of the first organization for a plurality of consumers, wherein the training data set comprises, for each consumer, identification information, and a set of behavioral attribute values, and

a second training data set of a second organization for a plurality of consumers, wherein the second training data set comprises, for each consumer, identification information, and actual behavioral attribute values for the unknown behavioral data attribute,

combining, by the second computing system, the first training data set and the second training data set into a joined data set by joining the one or more behavioral data attribute values of the first data set and the actual behavior attribute values for the unknown behavioral attribute of the second data set, for each consumer, where the common identifying information of the first data set and the common identifying information of the second set correspond,

calculating, by the second computing system, a vectorized training data set by applying the one or more vectorization rules to vectorize the set of behavioral attribute values for each consumer in the joined data set into a set of training vectors,

accepting, by the second computing system, a trainable model definition from an external source, wherein a trainable model defined by the trainable model definition accepts a training vector as input, and produces an estimated unknown behavioral data attribute as an output,

creating in one or more memories of the second computing system the defined trainable model, and

training, by the second computing system, the defined trainable model on the training vectors corresponding to a first subset of consumers in the joined data set to produce the trained model.

2. The method of claim 1, wherein training the model further comprises:

validating, by the second computing system, the trained model by transforming each training vector in a second subset of consumers in the joined data set into an estimated unknown behavioral data attribute, comparing the estimated unknown behavioral data attribute to the actual behavioral attribute value to produce a performance metric, and determining whether the performance metric falls within a predetermined range of acceptable performance values, wherein the first subset of consumers and the second subset of consumers is mutually exclusive.

3. The method of claim 1, wherein the trained model is a decision tree, and wherein the step of training the defined trainable model further comprises pruning each terminal node of the decisions tree wherein the terminal node includes fewer than a predetermined number of consumers, or a predetermined percentage of the total number of consumers.

4. The method of claim 1, wherein the set of behavioral attributes of the first training data set is a subset of available behavioral attributes, wherein the set of behavioral attributes are selected such that each behavioral attribute in the set of behavioral attributes is substantially statistically independent.

5. The method of claim 4, wherein each behavioral attribute in the first training data set was selected by:

evaluating the statistical independence of each pair of available behavioral attribute values for consumers in a consumer data set comprising available behavioral attribute values for a plurality of consumers,

aggregating the available behavioral attributes into substantially statistically independent subsets of behavioral attributes, and

selecting one behavioral attribute from each substantially statistically independent subset of behavioral attributes.

6. The method of claim 4, wherein the first organization has not been provided the second training data set of the second organization, and the joined data set.

7. The method of claim 4, wherein training the model further comprises the step of validating the trained model by calculating the sum of the estimated unknown consumer behavior for all consumers in a consumer data set, and comparing the sum to an estimated total behavior.

8. The method of claim 1, wherein training the model further comprises the step of destroying the joined data set.

9. A computing system for training a model to estimate an unknown consumer behavior, the computing system comprising:

one or more memories having computer readable computer instructions; and

one or more processors for executing the computer readable computer instructions to perform a method comprising;

calculating, by a first computing system, a vector representing a consumer'"'"'s behavior by applying one or more vectorization rules to a set of behavioral attribute values for the consumer in a data set of a first organization;

transforming the vector, by the first computing system, into an estimated unknown consumer behavioral attribute value by applying a trained model, wherein the model was trained by;

providing, to a second computing system separate from the first computing system;

a first training data set of the first organization for a plurality of consumers, wherein the training data set comprises, for each consumer, identification information, and a set of behavioral attribute values, and

a second training data set of a second organization for a plurality of consumers, wherein the second training data set comprises, for each consumer, identification information, and actual behavioral attribute values for the unknown behavioral data attribute,

combining, by the second computing system, the first training data set and the second training data set into a joined data set by joining the one or more behavioral data attribute values of the first data set and the actual behavior attribute values for the unknown behavioral attribute of the second data set, for each consumer, where the common identifying information of the first data set and the common identifying information of the second set correspond,

calculating, by the second computing system, a vectorized training data set by applying the one or more vectorization rules to vectorize the set of behavioral attribute values for each consumer in the joined data set into a set of training vectors,

accepting, by the second computing system, a trainable model definition from an external source, wherein a trainable model defined by the trainable model definition accepts a training vector as input, and produces an estimated unknown behavioral data attribute as an output,

creating in one or more memories of the second computing system the defined trainable model, and

training, by the second computing system, the defined trainable model on the training vectors corresponding to a first subset of consumers in the joined data set to produce the trained model.

10. The computing system of claim 9, wherein training the model further comprises:

validating, by the second computing system, the trained model by transforming each training vector in a second subset of consumers in the joined data set into an estimated unknown behavioral data attribute, comparing the estimated unknown behavioral data attribute to the actual behavioral attribute value to produce a performance metric, and determining whether the performance metric falls within a predetermined range of acceptable performance values, wherein the first subset of consumers and the second subset of consumers is mutually exclusive.

11. The computing system of claim 9, wherein the trained model is a decision tree, and wherein the step of training the defined trainable model further comprises pruning each terminal node of the decisions tree wherein the terminal node includes fewer than a predetermined number of consumers, or a predetermined percentage of the total number of consumers.

12. The computing system of claim 9, wherein the set of behavioral attributes of the first training data set is a subset of available behavioral attributes, wherein the set of behavioral attributes are selected such that each behavioral attribute in the set of behavioral attributes is substantially statistically independent.

13. The computing system of claim 12, wherein each behavioral attribute in the first training data set was selected by:

evaluating the statistical independence of each pair of available behavioral attribute values for consumers in a consumer data set comprising available behavioral attribute values for a plurality of consumers,

aggregating the available behavioral attributes into substantially statistically independent subsets of behavioral attributes, and

selecting one behavioral attribute from each substantially statistically independent subset of behavioral attributes.

14. The computing system of claim 9, wherein the first organization has not been provided the second training data set of the second organization, and the joined data set.

15. The computing system of claim 9, wherein training the model further comprises the step of validating the trained model by calculating the sum of the estimated unknown consumer behavior for all consumers in a consumer data set, and comparing the sum to an estimated total behavior.

16. The computing system of claim 9, wherein training the model further comprises the step of destroying the joined data set.

17. One or more non-transitory computer-readable storage media containing machine-readable computer instructions that, when executed by a computing system, performs a method for training a model to estimate an unknown consumer behavior, comprising the steps of:

calculating, by a first computing system, a vector representing a consumer'"'"'s behavior by applying one or more vectorization rules to a set of behavioral attribute values for the consumer in a data set of a first organization;

transforming the vector, by the first computing system, into an estimated unknown consumer behavioral attribute value by applying a trained model, wherein the model was trained by;

providing, to a second computing system separate from the first computing system;

a first training data set of the first organization for a plurality of consumers, wherein the training data set comprises, for each consumer, identification information, and a set of behavioral attribute values, and

a second training data set of a second organization for a plurality of consumers, wherein the second training data set comprises, for each consumer, identification information, and actual behavioral attribute values for the unknown behavioral data attribute,

combining, by the second computing system, the first training data set and the second training data set into a joined data set by joining the one or more behavioral data attribute values of the first data set and the actual behavior attribute values for the unknown behavioral attribute of the second data set, for each consumer, where the common identifying information of the first data set and the common identifying information of the second set correspond,

calculating, by the second computing system, a vectorized training data set by applying the one or more vectorization rules to vectorize the set of behavioral attribute values for each consumer in the joined data set into a set of training vectors,

accepting, by the second computing system, a trainable model definition from an external source, wherein a trainable model defined by the trainable model definition accepts a training vector as input, and produces an estimated unknown behavioral data attribute as an output,

creating in one or more memories of the second computing system the defined trainable model, and

training, by the second computing system, the defined trainable model on the training vectors corresponding to a first subset of consumers in the joined data set to produce the trained model.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein training the model further comprises:

validating, by the second computing system, the trained model by transforming each training vector in a second subset of consumers in the joined data set into an estimated unknown behavioral data attribute, comparing the estimated unknown behavioral data attribute to the actual behavioral attribute value to produce a performance metric, and determining whether the performance metric falls within a predetermined range of acceptable performance values, wherein the first subset of consumers and the second subset of consumers is mutually exclusive.

19. The one or more non-transitory computer-readable storage media of claim 17, wherein the trained model is a decision tree, and wherein the step of training the defined trainable model further comprises pruning each terminal node of the decisions tree wherein the terminal node includes fewer than a predetermined number of consumers, or a predetermined percentage of the total number of consumers.

20. The one or more non-transitory computer-readable storage media of claim 17, wherein the set of behavioral attributes of the first training data set is a subset of available behavioral attributes, wherein the set of behavioral attributes are selected such that each behavioral attribute in the set of behavioral attributes is substantially statistically independent.

21. The one or more non-transitory computer-readable storage media of claim 20, wherein each behavioral attribute in the first training data set was selected by:

evaluating the statistical independence of each pair of available behavioral attribute values for consumers in a consumer data set comprising available behavioral attribute values for a plurality of consumers,

aggregating the available behavioral attributes into substantially statistically independent subsets of behavioral attributes, and

selecting one behavioral attribute from each substantially statistically independent subset of behavioral attributes.

22. The one or more non-transitory computer-readable storage media of claim 17, wherein the first organization has not been provided the second training data set of the second organization, and the joined data set.

23. The one or more non-transitory computer-readable storage media of claim 17, wherein training the model further comprises the step of validating the trained model by calculating the sum of the estimated unknown consumer behavior for all consumers in a consumer data set, and comparing the sum to an estimated total behavior.

24. The one or more non-transitory computer-readable storage media of claim 17, wherein training the model further comprises the step of destroying the joined data set.