Data model generation using generative adversarial networks

• US 10,460,235 B1
• Filed: 10/04/2018
• Issued: 10/29/2019
• Est. Priority Date: 07/06/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Truong, Anh, Abdi Taghi Abad, Fardin, Goodsitt, Jeremy, Walters, Austin, Watson, Mark, Pham, Vincent, Key, Kate, Farivar, Reza, Taylor, Kenneth
• Primary Examiner(s): Chen, Alan
• Application Number:
  US16/151,385
• Time in Patent Office: 390 Days
• Field of Search: None
• CPC Class Codes: G06F 11/3608 using formal methods, e.g. model checking, abstract interpretation theorem proving G06N5/013
• CPC Class Codes: G06F 11/3628 of optimised code optimisation G06F8/443
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/2423 Interactive query statement specification based on a database schema
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 : Presentation of query results
• CPC Class Codes: G06F 16/906 : Clustering; Classification
• CPC Class Codes: G06F 16/93 : Document management systems
• CPC Class Codes: G06F 17/15 : Correlation function computation including computation of convolution operations arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00
• CPC Class Codes: G06F 17/16 : Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 17/18 : for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 : by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/213 : Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/214 : Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2148 : characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2193 : based on specific statistical tests
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 : Clustering techniques
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/2411 : based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/2415 : based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/285 : Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 18/40 : Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06F 21/552 : involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/60 : Protecting data
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/6254 : by anonymising data, e.g. decorrelating personal data from the owner's identification
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06F 40/117 : Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/166 : Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/20 : Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 8/71 : Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: G06F 9/541 : via adapters, e.g. between incompatible applications
• CPC Class Codes: G06F 9/547 : Remote procedure calls [RPC]; Web services
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 20/10 : using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 : Ensemble learning
• CPC Class Codes: G06N 3/04 : Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/047 : Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/06 : Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 : Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 3/094 : Adversarial learning
• CPC Class Codes: G06N 5/00 : Computing arrangements using knowledge-based models
• CPC Class Codes: G06N 5/01 : Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/02 : Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 : Inference or reasoning models
• CPC Class Codes: G06N 7/00 : Computing arrangements based on specific mathematical models
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/04 : Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06T 11/001 : Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/254 : involving subtraction of images
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 10/993 : Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 30/1985 : Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1491 : using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04N 21/23412 : for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/8153 : comprising still images, e.g. texture, background image

Abstract

Methods for generating data models using a generative adversarial network can begin by receiving a data model generation request by a model optimizer from an interface. The model optimizer can provision computing resources with a data model. As a further step, a synthetic dataset for training the data model can be generated using a generative network of a generative adversarial network, the generative network trained to generate output data differing at least a predetermined amount from a reference dataset according to a similarity metric. The computing resources can train the data model using the synthetic dataset. The model optimizer can evaluate performance criteria of the data model and, based on the evaluation of the performance criteria of the data model, store the data model and metadata of the data model in a model storage. The data model can then be used to process production data.

Claims

1. A method for generating data models, comprising:

receiving, by a model optimizer from an interface, a data model generation request;

provisioning, by the model optimizer, computing resources with a data model;

generating, by a dataset generator, a synthetic dataset for training the data model using a generative network of a generative adversarial network, the generative network trained to generate output data differing at least a predetermined amount from a reference dataset according to a similarity metric;

generating, by the model optimizer, at least one of a statistical correlation score between the synthetic dataset and the reference dataset, a data similarity score between the synthetic dataset and the reference dataset, and a data quality score for the synthetic dataset;

training, by the computing resources, the data model using the synthetic dataset, wherein training the data model using the synthetic dataset comprises determining that the synthetic dataset satisfies a criterion concerning the at least one of the statistical correlation score between the synthetic dataset and the reference dataset, the data similarity score between the synthetic dataset and the reference dataset, and the data quality score for the synthetic dataset;

evaluating, by the model optimizer, performance criteria of the data model;

storing, by the model optimizer in a model storage, the data model and metadata of the data model based on the evaluation of the performance criteria of the data model; and

processing production data using the data model.

2. The method of claim 1, wherein the similarity metric depends on a maximum distance or an average distance according to a distance measure between rows selected from the output data and row selected from the reference dataset.

3. The method of claim 1, wherein the generative network is trained to generate the output data with less than a predetermined proportion of duplicate elements.

4. The method of claim 1, wherein the generative network is trained to generate the output data with an output data schema matching a schema of the reference dataset.

5. The method of claim 1, wherein the method further comprises training the generative adversarial network using a loss function that penalizes generation of data differing from the reference dataset by less than the predetermined amount.

6. The method of claim 1, wherein the method further comprises:

receiving the reference dataset;

normalizing the reference dataset;

generating a synthetic training dataset using the generative network;

determining a similarity metric value according to the similarity metric using the normalized reference dataset and the synthetic training dataset;

updating a loss function that penalizes generation of data differing from the reference dataset by less than the predetermined amount using the similarity metric value; and

training the generative adversarial network using the normalized reference dataset, the synthetic training dataset, and the updated loss function.

7. The method of claim 6, wherein the generative network comprises a decoder network configured to generate decoder output data in a sample space having a first dimensionality from decoder input data in a code space having a second dimensionality less than the first dimensionality.

8. The method of claim 7, wherein generating the synthetic dataset for training the data model comprises:

identifying first points and second points in the sample space;

generating first corresponding points and second corresponding points in the code space using an encoder network corresponding to the decoder network, the first points, and the second points;

determining a first representative point based on the first corresponding points and a second representative point based on the second corresponding points;

determining a vector connecting the first representative point and the second representative point;

translating datapoints in the code space using the vector and a scaling factor; and

converting the translated datapoints into the sample space using the decoder network.

9. The method of claim 8, wherein the first representative point is a centroid or a medoid of the first corresponding points.

10. The method of claim 9, wherein generating the synthetic dataset for training the data model comprises:

identifying a first point and a second point in the sample space;

generating a first representative point and a second representative point in the code space using the first point, the second point, and an encoder network corresponding to the decoder network;

determining a vector connecting the first representative point and the second representative point;

generating an extreme point in the code space by sampling the code space along an extension of the vector beyond the second representative point; and

converting the extreme point in the code space into the sample space using the decoder network.

11. A cloud computing system for generating data models, comprising:

at least one processor; and

at least one non-transitory memory storing instructions that, when executed by the at least one processor, cause the cloud computing system to perform operations comprising;

receiving, by a model optimizer from an interface, a data model generation request;

provisioning, by the model optimizer, computing resources with a data model;

generating, by a generative network of a generative adversarial network, a synthetic dataset for training the data model;

generating, by the model optimizer, at least one of a statistical correlation score between the synthetic dataset and the reference dataset, a data similarity score between the synthetic dataset and the reference dataset, and a data quality score for the synthetic dataset;

training, by the computing resources, the data model using the synthetic dataset, wherein training the data model using the synthetic dataset comprises determining that the synthetic dataset satisfies a criterion concerning the at least one of the statistical correlation score between the synthetic dataset and the reference dataset, the data similarity score between the synthetic dataset and the reference dataset, and the data quality score for the synthetic dataset;

evaluating, by the model optimizer, performance criteria of the data model;

storing, by the model optimizer in a model storage, the data model and metadata of the data model based on the evaluation of the performance criteria of the data model; and

processing production data using the data model.

12. The system of claim 11, wherein:

the operations further comprise;

retrieving a reference dataset from a database, the reference dataset including categorical data;

generating a normalized training dataset by normalizing the categorical data; and

training the generative network using the normalized training dataset.

13. The system of claim 12, wherein normalizing the categorical data comprises converting the categorical data to numerical values within a predetermined range.

14. The system of claim 13, wherein:

the reference dataset includes at least one of missing values or not-a-number values; and

generating the normalized training dataset by normalizing the categorical data comprises converting the at least one of the missing values or the not-a-number values to corresponding predetermined numerical values outside the predetermined range.

15. The system of claim 11, wherein the generative network is configured to generate output data differing at least a predetermined amount from a reference dataset according to a similarity metric.

16. The system of claim 15, wherein the similarity metric depends on a covariance of numeric elements of the output data and a covariance of numeric elements of the reference dataset.

17. The system of claim 15, wherein the similarity metric depends on a univariate value distribution of an element of the output data and a univariate value distribution of an element of the reference dataset.

18. The system of claim 15, wherein the similarity metric depends on a joint probability distribution of elements in the output data and a joint probability distribution of elements in the reference dataset.

19. The system of claim 15, wherein the similarity metric depends on a number of rows of the output data that match rows of the reference dataset.