DISTRIBUTED PROCESSING OF DATA RECORDS

• US 20150120758A1
• Filed: 10/15/2014
• Published: 04/30/2015
• Est. Priority Date: 10/29/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cichosz, Pawel, Skowronski, Krzysztof, Dendek, Cezary, Draminski, Michal, Klopotek, Miezyslaw
• Granted Patent: US 10,482,154 B2
• Time in Patent Office: Days
• US Class Current: 707/748
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78

Abstract

Embodiments relate to distributed processing of data on a distributed database computer system. An aspect includes distributing rows of an arbitrary matrix to all of a plurality of processing units, wherein a number of the rows is equal to a number of the processing units, wherein one row of the arbitrary matrix is stored in each storage memory. Another aspect includes executing a first user defined function (UDF) on each processing unit, wherein a Cartesian product of each processing parameter matrix and the row of the arbitrary matrix is calculated on each processing unit and the matrix set is stored in the processor memory of each processing unit; and executing a second UDF on each processing unit having at least one data record after the executing of the first UDF, wherein all data records stored in the storage memory of the each processing unit are processed one by one.

Claims

1. A computer implemented method for distributed processing of data on a distributed database computer system, the method comprising:

distributing rows of an arbitrary matrix to all of a plurality of processing units, wherein a number of the rows is equal to a number of the processing units, wherein a matrix set is used to calculate a value set corresponding to each data record by using said data record as input, the matrix set comprising at least one processing parameter matrix, the value set comprising at least one calculation value, the distributed database computer system comprising the plurality of processing units connected in a share-nothing parallel processing architecture, wherein each processing unit comprises a processor of the each processing unit, a processor memory of the each processing unit, and a storage memory of the each processing unit, wherein the arbitrary matrix is stored in the distributed data base in a way that one row of the arbitrary matrix is stored in each storage memory, wherein the data records and the matrix set are stored in a distributed database using the storage memories, each processor is being operable for executing user defined functions (UDFs), calculating the value set corresponding to only one data record at a time, executing transaction processing, storing data in the processor memory, and using the data stored in the processor memory for execution of the UDFs within a framework of one transaction; and

performing transaction processing in a framework of one transaction by;

executing a first UDF on each processing unit, wherein a Cartesian product of the each processing parameter matrix and the row of the arbitrary matrix is calculated on each processing unit and as a result thereof the matrix set is stored in the processor memory of each processing unit; and

executing a second UDF on each processing unit having at least one data record after the executing of the first UDF, wherein a number of repetitive executions of the second UDF on each processing unit is equal to the number of the data records stored in the storage memory of the each processing unit and all data records stored in the storage memory of the each processing unit are processed one by one, wherein the value set corresponding to the data record is calculated using the matrix set stored in the processor memory of said respective processing unit.

2. The method of claim 1, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of a link function of a generalized linear model, wherein the matrix set comprises the following processing parameter matrices:

a metadata matrix, a predictor-to-parameter correlation matrix, and a beta coefficient matrix, wherein the metadata matrix determines a type of the link function and a distribution function used in said link function, the predictor-to-parameter correlation matrix determining correlation between each calculation value of the value set and the corresponding data record in said link function, wherein the predictor-to-parameter correlation matrix comprises rows of the predictor-to-parameter correlation matrix and each row of the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of the beta coefficient matrix in said link function.

3. The method of claim 1, wherein the distributed database is a Standard Query Language (SQL) distributed database, wherein the first UDF is one or more first SQL UDFs, wherein the second UDF is a second SQL UDF, wherein each first SQL UDF is performed by cross joining of the arbitrary matrix with one or more processing parameter matrices within a SQL query of the each first SQL UDF, wherein the second SQL UDF is performed by querying the data record in the distributed database within a SQL query of the second SQL UDF and after the querying of the data record in the distributed database causing the processor of the processing unit to calculate the value set corresponding to said data record within said SQL query of the second SQL UDF, wherein said processing unit comprises the storage memory comprising said data record.

4. The method of claim 1, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of link functions, wherein each link function is a link function of a unique generalized linear model, wherein each link function generates the corresponding portion of the calculation values of the value set, wherein the matrix set comprises the flowing sets of the processing parameter matrixes:

a metadata matrix set, a predictor-to-parameter correlation matrix set, and a beta coefficient matrix set, wherein the metadata matrix set comprises at least one metadata matrix, wherein the predictor-to-parameter correlation matrix set comprises at least one predictor-to-parameter correlation matrix, wherein the beta coefficient matrix set comprises at least one beta coefficient matrix, wherein each generalized linear model has a unique combination of one metadata matrix of the metadata matrix set, one predictor-to-parameter correlation matrix of the predictor-to-parameter correlation matrix set, and one beta coefficient matrix of the beta coefficient matrix set, wherein the metadata matrix of each generalized linear model determines a type of the link function of the each generalized linear model and a distribution function used in the link function of the each generalized linear model, wherein the predictor-to-parameter correlation matrix of each generalized linear model determines correlation between each calculation value of the portion of the calculation values and the corresponding data record in the link function of the each generalized linear model, wherein the predictor-to-parameter correlation matrix of each generalized linear model comprises rows of said the predictor-to-parameter correlation matrix and each row of said the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of a beta coefficient matrix of the each generalized linear model in the link function of the each generalized linear model.

5. The method of claim 1, wherein the data records are linear arrays of data values.

6. The method of claim 1, wherein at least one processing parameter matrix is a linear array of processing parameter values.

7. A computer program product for distributed processing of data on a distributed database computer system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

distributing rows of an arbitrary matrix to all of a plurality of processing units, wherein a number of the rows is equal to a number of the processing units, wherein a matrix set is used to calculate a value set corresponding to each data record by using said data record as input, the matrix set comprising at least one processing parameter matrix, the value set comprising at least one calculation value, the distributed database computer system comprising the plurality of processing units connected in a share-nothing parallel processing architecture, wherein each processing unit comprises a processor of the each processing unit, a processor memory of the each processing unit, and a storage memory of the each processing unit, wherein the arbitrary matrix is stored in the distributed data base in a way that one row of the arbitrary matrix is stored in each storage memory, wherein the data records and the matrix set are stored in a distributed database using the storage memories, each processor is being operable for executing user defined functions (UDFs), calculating the value set corresponding to only one data record at a time, executing transaction processing, storing data in the processor memory, and using the data stored in the processor memory for execution of the UDFs within a framework of one transaction; and

performing transaction processing in a framework of one transaction by;

executing a first UDF on each processing unit, wherein a Cartesian product of the each processing parameter matrix and the row of the arbitrary matrix is calculated on each processing unit and as a result thereof the matrix set is stored in the processor memory of each processing unit; and

executing a second UDF on each processing unit having at least one data record after the executing of the first UDF, wherein a number of repetitive executions of the second UDF on each processing unit is equal to the number of the data records stored in the storage memory of the each processing unit and all data records stored in the storage memory of the each processing unit are processed one by one, wherein the value set corresponding to the data record is calculated using the matrix set stored in the processor memory of said respective processing unit.

8. The computer program product of claim 7, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of a link function of a generalized linear model, wherein the matrix set comprises the following processing parameter matrices:

a metadata matrix, a predictor-to-parameter correlation matrix, and a beta coefficient matrix, wherein the metadata matrix determines a type of the link function and a distribution function used in said link function, the predictor-to-parameter correlation matrix determining correlation between each calculation value of the value set and the corresponding data record in said link function, wherein the predictor-to-parameter correlation matrix comprises rows of the predictor-to-parameter correlation matrix and each row of the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of the beta coefficient matrix in said link function.

9. The computer program product of claim 7, wherein the distributed database is a Standard Query Language (SQL) distributed database, wherein the first UDF is one or more first SQL UDFs, wherein the second UDF is a second SQL UDF, wherein each first SQL UDF is performed by cross joining of the arbitrary matrix with one or more processing parameter matrices within a SQL query of the each first SQL UDF, wherein the second SQL UDF is performed by querying the data record in the distributed database within a SQL query of the second SQL UDF and after the querying of the data record in the distributed database causing the processor of the processing unit to calculate the value set corresponding to said data record within said SQL query of the second SQL UDF, wherein said processing unit comprises the storage memory comprising said data record.

10. The computer program product of claim 7, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of link functions, wherein each link function is a link function of a unique generalized linear model, wherein each link function generates the corresponding portion of the calculation values of the value set, wherein the matrix set comprises the flowing sets of the processing parameter matrixes:

a metadata matrix set, a predictor-to-parameter correlation matrix set, and a beta coefficient matrix set, wherein the metadata matrix set comprises at least one metadata matrix, wherein the predictor-to-parameter correlation matrix set comprises at least one predictor-to-parameter correlation matrix, wherein the beta coefficient matrix set comprises at least one beta coefficient matrix, wherein each generalized linear model has a unique combination of one metadata matrix of the metadata matrix set, one predictor-to-parameter correlation matrix of the predictor-to-parameter correlation matrix set, and one beta coefficient matrix of the beta coefficient matrix set, wherein the metadata matrix of each generalized linear model determines a type of the link function of the each generalized linear model and a distribution function used in the link function of the each generalized linear model, wherein the predictor-to-parameter correlation matrix of each generalized linear model determines correlation between each calculation value of the portion of the calculation values and the corresponding data record in the link function of the each generalized linear model, wherein the predictor-to-parameter correlation matrix of each generalized linear model comprises rows of said the predictor-to-parameter correlation matrix and each row of said the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of a beta coefficient matrix of the each generalized linear model in the link function of the each generalized linear model.

11. The computer program product of claim 7, wherein the data records are linear arrays of data values.

12. The computer program product of claim 7, wherein at least one processing parameter matrix is a linear array of processing parameter values.

13. A computer system distributed processing of data on a distributed database computer system, the system comprising:

a memory; and

a processor, communicatively coupled to said memory, the computer system configured to perform a method comprising;

distributing rows of an arbitrary matrix to all of a plurality of processing units, wherein a number of the rows is equal to a number of the processing units, wherein a matrix set is used to calculate a value set corresponding to each data record by using said data record as input, the matrix set comprising at least one processing parameter matrix, the value set comprising at least one calculation value, the distributed database computer system comprising the plurality of processing units connected in a share-nothing parallel processing architecture, wherein each processing unit comprises a processor of the each processing unit, a processor memory of the each processing unit, and a storage memory of the each processing unit, wherein the arbitrary matrix is stored in the distributed data base in a way that one row of the arbitrary matrix is stored in each storage memory, wherein the data records and the matrix set are stored in a distributed database using the storage memories, each processor is being operable for executing user defined functions (UDFs), calculating the value set corresponding to only one data record at a time, executing transaction processing, storing data in the processor memory, and using the data stored in the processor memory for execution of the UDFs within a framework of one transaction; and

performing transaction processing in a framework of one transaction by;

executing a first UDF on each processing unit, wherein a Cartesian product of the each processing parameter matrix and the row of the arbitrary matrix is calculated on each processing unit and as a result thereof the matrix set is stored in the processor memory of each processing unit; and

executing a second UDF on each processing unit having at least one data record after the executing of the first UDF, wherein a number of repetitive executions of the second UDF on each processing unit is equal to the number of the data records stored in the storage memory of the each processing unit and all data records stored in the storage memory of the each processing unit are processed one by one, wherein the value set corresponding to the data record is calculated using the matrix set stored in the processor memory of said respective processing unit.

14. The system of claim 13, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of a link function of a generalized linear model, wherein the matrix set comprises the following processing parameter matrices:

a metadata matrix, a predictor-to-parameter correlation matrix, and a beta coefficient matrix, wherein the metadata matrix determines a type of the link function and a distribution function used in said link function, the predictor-to-parameter correlation matrix determining correlation between each calculation value of the value set and the corresponding data record in said link function, wherein the predictor-to-parameter correlation matrix comprises rows of the predictor-to-parameter correlation matrix and each row of the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of the beta coefficient matrix in said link function.

15. The system of claim 13, wherein the distributed database is a Standard Query Language (SQL) distributed database, wherein the first UDF is one or more first SQL UDFs, wherein the second UDF is a second SQL UDF, wherein each first SQL UDF is performed by cross joining of the arbitrary matrix with one or more processing parameter matrices within a SQL query of the each first SQL UDF, wherein the second SQL UDF is performed by querying the data record in the distributed database within a SQL query of the second SQL UDF and after the querying of the data record in the distributed database causing the processor of the processing unit to calculate the value set corresponding to said data record within said SQL query of the second SQL UDF, wherein said processing unit comprises the storage memory comprising said data record.

16. The system of claim 13, wherein the matrix set is used to calculate the value set corresponding to each data record by using said data record as input of link functions, wherein each link function is a link function of a unique generalized linear model, wherein each link function generates the corresponding portion of the calculation values of the value set, wherein the matrix set comprises the flowing sets of the processing parameter matrixes:

a metadata matrix set, a predictor-to-parameter correlation matrix set, and a beta coefficient matrix set, wherein the metadata matrix set comprises at least one metadata matrix, wherein the predictor-to-parameter correlation matrix set comprises at least one predictor-to-parameter correlation matrix, wherein the beta coefficient matrix set comprises at least one beta coefficient matrix, wherein each generalized linear model has a unique combination of one metadata matrix of the metadata matrix set, one predictor-to-parameter correlation matrix of the predictor-to-parameter correlation matrix set, and one beta coefficient matrix of the beta coefficient matrix set, wherein the metadata matrix of each generalized linear model determines a type of the link function of the each generalized linear model and a distribution function used in the link function of the each generalized linear model, wherein the predictor-to-parameter correlation matrix of each generalized linear model determines correlation between each calculation value of the portion of the calculation values and the corresponding data record in the link function of the each generalized linear model, wherein the predictor-to-parameter correlation matrix of each generalized linear model comprises rows of said the predictor-to-parameter correlation matrix and each row of said the predictor-to-parameter correlation matrix is used in conjunction with a corresponding beta coefficient of a beta coefficient matrix of the each generalized linear model in the link function of the each generalized linear model.

17. The system of claim 13, wherein the data records are linear arrays of data values.

18. The system of claim 13, wherein at least one processing parameter matrix is a linear array of processing parameter values.