Method for generating synthetic data sets at scale with non-redundant partitioning

• US 10,891,311 B2
• Filed: 10/14/2016
• Issued: 01/12/2021
• Est. Priority Date: 10/14/2016
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Vyas, Jay, Nowling, Ronald, Chen, Huamin
• Primary Examiner(s): Morrison, Jay A
• Application Number:
  US15/294,142
• Publication Number:
  US 20180107729A1
• Time in Patent Office: 1,551 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

An example system includes a first machine and a second machine, a clustering module, and a training module. The clustering module receives a plurality of data sets, each including attributes. The clustering module partitions the plurality of data sets into a first clustered data set and a second clustered data set. Each data set of the plurality of data sets is partitioned. The training module assigns a first stochastic model to the first clustered data set and a second stochastic model to the second clustered data set. The first machine selects the first clustered data set and the first stochastic model and generates a first synthetic data set having generated data for each one of the attributes. The second machine selects the second clustered data set and the second stochastic model and generates a second synthetic data set having generated data for each one of the attributes.

Claims

1. A method comprising:

receiving, by a clustering module, a plurality of data sets, wherein each data set of the plurality of data sets includes a plurality of attributes;

partitioning, by the clustering module, the plurality of data sets into a plurality of clustered data sets including at least a first clustered data set and a second clustered data set, wherein each data set of the plurality of data sets is partitioned into one of the plurality of clustered data sets;

assigning, by a training module, a respective stochastic model to each respective clustered data set of the plurality of clustered data sets including;

assigning a first stochastic model to the first clustered data set, and

assigning a second stochastic model to the second clustered data set;

selecting, by a first machine including a first memory and one or more processors in communication with the first memory, the first clustered data set and the first stochastic model;

selecting, by a second machine that is different from the first machine, the second machine including a second memory and one or more processors in communication with the second memory, the second clustered data set and the second stochastic model;

generating, by the first machine with the first stochastic model, a first synthetic data set, wherein the first synthetic data set has generated data for each one of the plurality of attributes;

generating, by the second machine with the second stochastic model, a second synthetic data set, wherein the second synthetic data set has generated data for each one of the plurality of attributes; and

testing at least one of an application and a database using each of the first synthetic data set and the second synthetic data set.

2. The method of claim 1, wherein, responsive to the first machine selecting the first clustered data set and the first stochastic model, the first stochastic model is locked out, such that the second machine is unable to use the first stochastic model to generate the second synthetic data set.

3. The method of claim 2, wherein lockout of the first stochastic model is performed by a fencing token.

4. The method of claim 2, wherein an indication of the first stochastic model'"'"'s locked out status is recorded in a data store.

5. The method of claim 2, wherein upon selecting the first clustered data set and the first stochastic model, the first machine is granted a lease, wherein the lease has an expiration time.

6. The method of claim 5, wherein responsive to the lease expiring at the expiration time, the first stochastic model is unlocked, such that the first stochastic model may be selected by a third machine to generate a third synthetic data set.

7. The method of claim 1, wherein the training module creates the first stochastic model and the second stochastic model.

8. The method of claim 1, wherein each of the first stochastic model and the second stochastic model are Markov models.

9. The method of claim 1, wherein the first synthetic data set and the second synthetic data set are simultaneously generated in parallel.

10. The method of claim 9, wherein each respective stochastic model simultaneously generating data in parallel generates different classes of data, such that only one respective stochastic model generates each class of data at any point in time.

11. The method of claim 1, wherein each of the first machine and the second machine are remotely located physical computers.

12. The method of claim 1, wherein each of the first machine and the second machine are virtual machines.

13. The method of claim 1, wherein testing of the database includes indexing of the database using each of the first synthetic data set and the second synthetic data set.

14. A system comprising:

a first machine including a first memory and one or more processors in communication with the first memory;

a second machine that is different from the first machine, the second machine including a second memory and one or more processors in communication with the second memory;

a clustering module, wherein the clustering module;

receives a plurality of data sets, wherein each data set of the plurality of data sets includes a plurality of attributes; and

partitions the plurality of data sets into a plurality of clustered data sets including at least a first clustered data set and a second clustered data set, wherein each data set of the plurality of data sets is partitioned into one of the plurality of clustered data sets; and

a training module, wherein the training module;

assigns a respective stochastic model to each respective clustered data set of the plurality of clustered data sets including;

assigning a first stochastic model to the first clustered data set, and

assigning a second stochastic model to the second clustered data set;

wherein the first machine;

selects the first clustered data set and the first stochastic model; and

generates, with the first stochastic model, a first synthetic data set, wherein the first synthetic data set has generated data for each one of the plurality of attributes; and

wherein the second machine;

selects the second clustered data set and the second stochastic model; and

generates, with the second stochastic model, a second synthetic data set, wherein the second synthetic data set has generated data for each one of the plurality of attributes,

such that at least one of an application and a database is tested using each of the first synthetic data set and the second synthetic data set.

15. The system of claim 14, wherein, responsive to the first machine selecting the first clustered data set and the first stochastic model, the first stochastic model is locked out, such that the second machine is unable to use the first stochastic model to generate the second synthetic data set.

16. The system of claim 15, wherein upon selecting the first clustered data set and the first stochastic model, the first machine is granted a lease, wherein the lease has an expiration time.

17. The system of claim 16, wherein responsive to the lease expiring at the expiration time, the first stochastic model is unlocked, such that the first stochastic model may be selected by a third machine to generate a third synthetic data set.

18. The system of claim 14, wherein each of the first stochastic model and the second stochastic model are Markov models.

19. A computer readable non-transitory storage medium comprising executable instructions that, when executed, cause

a clustering module to:

receive a plurality of data sets, wherein each data set of the plurality of data sets includes a plurality of attributes; and

partition the plurality of data sets into a plurality of clustered data sets including at least a first clustered data set and a second clustered data set, wherein each data set of the plurality of data sets is partitioned into one of the plurality of clustered data sets;

a training module to;

assign a respective stochastic model to each respective clustered data set of the plurality of clustered data sets including;

assigning a first stochastic model to the first clustered data set, and

assigning a second stochastic model to the second clustered data set;

a first machine including a first memory and one or more processors in communication with the first memory to;

select the first clustered data set and the first stochastic model; and

generate, with the first stochastic model, a first synthetic data set, wherein the first synthetic data set has generated data for each one of the plurality of attributes; and

a second machine that is different from the first machine, the second machine including a first memory and one or more processors in communication with the first memory to;

select the second clustered data set and the second stochastic model; and

generate, with the second stochastic model, a second synthetic data set, wherein the second synthetic data set has generated data for each one of the plurality of attributes,

such that at least one of an application and a database is tested using each of the first synthetic data set and the second synthetic data set.