Systems and methods for general aggregation of characteristics and key figures

• US 20060069632A1
• Filed: 09/30/2005
• Published: 03/30/2006
• Est. Priority Date: 09/30/2004
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Baumann, Marcus, Kahn, Markus
• Granted Patent: US 7,596,520 B2
• Time in Patent Office: Days
• US Class Current: 705/35
• CPC Class Codes: G06Q 20/108 Remote banking, e.g. home banking
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes
• CPC Class Codes: G06Q 40/02 Banking, e.g. interest calculation or account maintenance credit or loans G06Q40/03
• CPC Class Codes: G06Q 40/03 Credit; Loans; Processing thereof
• CPC Class Codes: G06Q 40/06 Asset management; Financial planning or analysis

Abstract

Computer-implemented methods, computer systems, and computer programs product are provided for automated generic and parallel aggregation of characteristics and key figures of unsorted mass data being of specific economic interest, particularly associated with financial institutions, and with financial affairs in banking practice. The parallel aggregation may reduce the amount of data for a customer defined granularity for the purpose of facilitating the handling of raw data related to all areas of credit risk management in banking practice. Moreover, the computing power of software and the software performance run time, respectively, may be improved in the case of mass data.

Claims

1. A computer-implemented method for automated generic and parallel aggregation of characteristics and key figures of mass data, the mass data including M records from a single database of a single data source or from different databases of different data sources, particularly associated with financial institutions and with financial affairs in banking practice, and further including sets of rows and sets of columns, each row corresponding to a record, and the columns including fields of predetermined granularity characteristics and fields of predetermined key figures, wherein the aggregation reduces the amount of data to N≦

M records for a customer defined granularity, the method comprising the following steps;

receiving the mass data from a single database of a single data source or from different databases of different data sources associated with banking practice;

selecting predetermined granularity characteristics and predetermined key figures, and selecting predetermined aggregation operations to be carried out by the processing means of a data processing system;

reading input data from a single database of a single data source or from different databases of different data sources into the processing means of a data processing system;

preparing the input data as data packages being of the size M_p in a preparational step before the aggregation starts;

processing the data packages being of the size M_p in a parallel process by identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the records in each data package for key figures by using aggregation operations; and

following the aggregation, saving the results of each data package.

2. The method of claim 1, wherein the aggregation is computed for the pre-determined granularity characteristics and/or predetermined key figures using predetermined aggregation operations selected from a function pool and/or costumer defined aggregation operations to be defined by input means using the predetermined aggregation operations.

3. The method of claim 1, wherein the aggregation is computed for costumer defined granularity characteristics and/or costumer defined key figures that are to be defined by input means using the predetermined aggregation operations selected from a function pool and using the predetermined aggregation operations and/or the costumer defined aggregation operations.

4. The method of claim 1, wherein the data packages being of the size M_p are processed within a loop i=i+1 comprising the steps of:

a first parallel process for identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the data records in each data package for key figures by using aggregation operations;

thereby reducing the amount of data records to N_a≦

M_p; and

following the aggregation, saving the results of each data package in a local database and storing the size and the key of the first and the last record of each data package in a global database; and

subsequently checking the termination criterion for the loop i=i+1 (“

not in parallel”

) being;

if key_pos1,xε

(key_pos1,y;

key<sub>pos max;

y</sub>) then continue else terminate, wherein pos1 illustrates the first position of a data package, posmax illustrates the last position of a data package, and x, y illustrates the number of data package, and if the conditional criterion is not accomplished for all combinations (x, y), meaning that the conditional inquiry is true, thereby comparing the key of the first record of each data package with the first and the last record of all the rest of packages, assigning the aggregated packages to a second parallel process for merging adjacent data packages so as to rebuild new data packages, wherein adjacent packages are those packages with keys of the first record which are closest together and have violated the termination criterion, then storing the merged packages to a local database, and subsequently assigning the merged data packages again to the above mentioned first parallel process for reorganizing and sorting the new data packages, and thereafter aggregating the new data packages for key figures by using aggregation operations, and following the aggregation, after each loop cycle checking the termination criterion for the loop i=i+1 for all combinations (x, y) anew, repeatedly executing the loop i=i+1 while the termination criterion for the loop is not accomplished, else after accomplishing the criterion, i.e. all the data packages are disjoint with regard to the granularity characteristics, terminating the loop i=i+1.

5. The method of claim 1, wherein the data packages being of the size M_p are processed within a loop i=i+1 comprising the steps of:

a first parallel process for identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the data records in each data package for key figures by using aggregation operations;

thereby reducing the amount of data records to N_a≦

M_p, and following the aggregation, splitting the aggregated data packages into several smaller data sub packages being of the size N_sp and saving the results of each sub data package in a local database;

storing the size and the key of the first and the last record of each sub data package to a global database; and

subsequently identifying adjacent packages based on these small sub data packages by checking (“

not in parallel”

) the termination criterion for the loop i=i+1 being;

if key_pos1,xε

(key_pos1,y;

key<sub>pos max;

y</sub>) then continue else terminate, wherein pos1 illustrates the first position of a data package, posmax illustrates the last position of a data package, and x, y illustrates the number of data package, and if the conditional criterion is not accomplished for all combinations (x, y), meaning that the conditional inquiry is true, thereby comparing the key of the first record of each sub data package with the first and the last record of all the rest of sub data packages, assigning the sub data packages to a second parallel process for merging adjacent sub data packages so as to rebuild new data packages, wherein adjacent sub data packages are those data packages with keys of the first record which are closest together and have violated the termination criterion, and wherein by merging the sub data packages the original package size N is restored;

then storing the new data packages to a local database; and

subsequently assigning the new data packages again to the above mentioned first parallel process for reorganizing and sorting; and

thereafter aggregating the new data packages for key figures by using aggregation operations; and

following the aggregation, after each loop cycle splitting the aggregated data packages again into several smaller sub data packages and saving the results of each sub data package in a local database;

storing the size and the key of the first and the last record of each sub data package to a global database; and

subsequently identifying adjacent packages again based on these small packages by checking (“

not in parallel”

) the termination criterion for the loop i=i+1 for all combinations (x, y) anew, repeatedly executing the loop i=i+1 while the termination criterion for the loop is not accomplished, else after accomplishing the criterion, i.e. all the data packages are disjoint with regard to the granularity characteristics, terminating the loop i=i+1.

6. The method of claim 4, wherein ultimately conducting an additional calculation step for enriching aggregated data packages, and subsequently saving the data packages to a global result database.

7. The method of claim 1, further comprising the steps of:

enriching the prepared data packages in a parallel pre-processing step via secondary data source or data sources before the parallel aggregation starts; and

saving the results to a local database.

8. The method of claim 6, further comprising the steps of:

enriching the aggregated data packages in a parallel post-processing step via secondary data source or data sources following the parallel aggregation; and

saving the results to a global result database.

9. The method of claim 1, wherein the data packages are processed within jobs, each of the jobs including a plurality of data packages.

10. The method of claim 9, wherein one job or a plurality of jobs are processed in a parallel processing mode using a single data processor.

11. The method of claim 9, wherein one job or a plurality of jobs are processed in a parallel processing mode using a network of data processors.

12. The method of claim 11, wherein the network of data processors is a Local Area Network (LAN), Wide Area Network (WAN), intranet or internet.

13. The method of claims 1, wherein the data packages are processed within jobs, and wherein the jobs are processed in a parallel processing mode using a single data processor, thereby aggregating and merging the data packages of a job sequentially.

14. The method of claim 1, wherein the data packages are processed within jobs, and wherein the jobs are processed in a parallel processing mode using a network of data processors, thereby aggregating and merging the data packages of a job sequentially.

15. A computer system configured to perform automated generic and parallel aggregation of characteristics and key figures of mass data, the mass data including M records from a single database of a single data source or from different databases of different data sources, particularly associated with financial institutions and with financial affairs in banking practice, and further including sets of rows and sets of columns, each row corresponding to a record, and the columns including fields of predetermined granularity characteristics and fields of predetermined key figures, wherein the aggregation reduces the amount of data to N≦

M records for a customer defined granularity, comprising;

means for receiving the mass data from a single database of a single data source or from different databases of different data sources associated with banking practice;

means for selecting predetermined granularity characteristics and predetermined key figures, and selecting predetermined aggregation operations to be carried out by the processing means of a data processing system;

means for reading input data from a single database of a single data source or from different databases of different data sources into the processing means of a data processing system;

means for preparing the input data as data packages being of the size M_p in a preparational step before the aggregation starts;

means for processing the data packages being of the size M_p in a parallel process by identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the records in each data package for key figures by using aggregation operations; and

means for, following the aggregation, saving the results of each data package.

16. A computer system of claim 15, wherein the processing means is configured to process the data packages being of the size M_p within a loop i=i+1 and includes the steps of:

a first parallel process for identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the data records in each data package for key figures by using aggregation operations;

thereby reducing the amount of data records to N_a≦

M_p; and

following the aggregation, saving the results of each data package in a local database and storing the size and the key of the first and the last record of each data package in a global database; and

subsequently checking the termination criterion for the loop i=i+1 (“

not in parallel”

) being;

if key_pos1,xε

(key_pos1,y;

key<sub>pos max;

y</sub>) then continue else terminate, wherein pos1 illustrates the first position of a data package, posmax illustrates the last position of a data package, and x, y illustrates the number of data package, and if the conditional criterion is not accomplished for all combinations (x, y), meaning that the conditional inquiry is true, thereby comparing the key of the first record of each data package with the first and the last record of all the rest of packages, assigning the aggregated packages to a second parallel process for merging adjacent data packages so as to rebuild new data packages, wherein adjacent packages are those packages with keys of the first record which are closest together and have violated the termination criterion, then storing the merged packages to a local database, and subsequently assigning the merged data packages again to the above mentioned first parallel process for reorganizing and sorting the new data packages, and thereafter aggregating the new data packages for key figures by using aggregation operations, and following the aggregation, after each loop cycle checking the termination criterion for the loop i=i+1 for all combinations (x, y) anew, repeatedly executing the loop i=i+1 while the termination criterion for the loop is not accomplished, else after accomplishing the criterion, i.e. all the data packages are disjoint with regard to the granularity characteristics, terminating the loop i=i+1.

17. A computer system of claim 15, wherein the processing means are configured to process the data packages being of the size M_p within a loop i=i+1 and includes the steps of:

a first parallel process for identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the data records in each data package for key figures by using aggregation operations;

thereby reducing the amount of data records to N_a≦

M_p and following the aggregation, splitting the aggregated packages into several smaller sub packages and saving the results of each sub data package in a local database;

storing the size and the key of the first and the last record of each sub data package to a global database; and

subsequently identifying adjacent packages based on these small sub data packages by checking (“

not in parallel”

) the termination criterion for the loop i=i+1 being;

if key_pos1,xε

(key_pos1,y;

key<sub>pos max;

y</sub>) then continue else terminate, wherein pos1 illustrates the first position of a data package, posmax illustrates the last position of a data package, and x, y illustrates the number of data package, and if the conditional criterion is not accomplished for all combinations (x, y), meaning that the conditional inquiry is true, thereby comparing the key of the first record of each sub data package with the first and the last record of all the rest of sub data packages, assigning the sub data packages to a second parallel process for merging adjacent sub data packages so as to rebuild new data packages, wherein adjacent sub data packages are those data packages with keys of the first record which are closest together and have violated the termination criterion, and wherein by merging the sub data packages the original package size N is restored;

then storing the new data packages to a local database; and

subsequently assigning the new data packages again to the above mentioned first parallel process for reorganizing and sorting; and

thereafter aggregating the new data packages for key figures by using aggregation operations; and

following the aggregation, after each loop cycle splitting the aggregated data packages again into several smaller sub data packages and saving the results of each sub data package in a local database;

storing the size and the key of the first and the last record of each sub data package to a global database; and

subsequently identifying adjacent packages again based on these small packages by checking (“

not in parallel”

) the termination criterion for the loop i=i+1 for all combinations (x, y) anew, repeatedly executing the loop i=i+1 while the termination criterion for the loop is not accomplished, else after accomplishing the criterion, i.e. all the data packages are disjoint with regard to the granularity characteristics, terminating the loop i=i+1.

18. A computer system of claim 16, wherein the processing means is further configured to ultimately conduct an calculation step for enriching data packages, and wherein the storage means are further configured to subsequently save the data packages to a global result database.

19. A computer system of claim 15, wherein the processing means is further configured to ultimately enrich the prepared data packages in a parallel pre-processing step via secondary data source or data sources before the parallel aggregation starts, and wherein the storage means are further configured to save the results to the local database.

20. A computer system of claim 18, wherein the processing means is further configured to ultimately enrich the aggregated data packages in a parallel post-processing step via secondary data source or data sources following the parallel aggregation, and wherein the storage means are further configured to save the results to the global database.

21. A computer program product having a plurality of instructions for causing processing means of a computer system to execute steps of:

receiving the mass data from a single database of a single data source or from different databases of different data sources associated with banking practice;

selecting predetermined granularity characteristics and predetermined key figures, and selecting predetermined aggregation operations to be carried out by the processing means of a data processing system;

reading input data from a single database of a single data source or from different databases of different data sources into the processing means of a data processing system;

preparing the input data as data packages being of the size M_p in a preparational step before the aggregation starts;

processing the data packages being of the size M_p in a parallel process by identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the records in each data package for key figures by using aggregation operations; and

following the aggregation, saving the results of each data package.

22. The computer program product of claim 21, wherein the program comprises instructions for processing the data packages being of the size M_p within a loop i=i+1 including the steps of:

a first parallel process for identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the data records in each data package for key figures by using aggregation operations;

thereby reducing the amount of data records to N_a≦

M_p; and

following the aggregation, saving the results of each data package in a local database and storing the size and the key of the first and the last record of each data package in a global database; and

subsequently checking the termination criterion for the loop i=i+1 (“

not in parallel”

) being;

if key_pos1,xε

(key_pos1,y;

key<sub>pos max;

y</sub>) then continue else terminate, wherein pos1 illustrates the first position of a data package, posmax illustrates the last position of a data package, and x, y illustrates the number of data package, and if the conditional criterion is not accomplished for all combinations (x, y), meaning that the conditional inquiry is true, thereby comparing the key of the first record of each data package with the first and the last record of all the rest of packages, assigning the aggregated packages to a second parallel process for merging adjacent data packages so as to rebuild new data packages, wherein adjacent packages are those packages with keys of the first record which are closest together and have violated the termination criterion, then storing the merged packages to a local database, and subsequently assigning the merged data packages again to the above mentioned first parallel process for reorganizing and sorting the new data packages, and thereafter aggregating the new data packages for key figures by using aggregation operations, and following the aggregation, after each loop cycle checking the termination criterion for the loop i=i+1 for all combinations (x, y) anew, repeatedly executing the loop i=i+1 while the termination criterion for the loop is not accomplished, else after accomplishing the criterion, i.e. all the data packages are disjoint with regard to the granularity characteristics, terminating the loop i=i+1.

23. The computer program product of claim 21, wherein the program comprises instructions for processing the data packages being of the size M_p within a loop i=i+1 including the steps of:

a first parallel process for identifying the granularity characteristics, thereby identifying unique granularity levels i;

sorting the records of each data package for a given order of granularity characteristics of the customized granularity; and

subsequently aggregating the data records in each data package for key figures by using aggregation operations;

thereby reducing the amount of data records to N_a≦

M_p, and following the aggregation, splitting the aggregated data packages into several smaller data sub packages being of the size N_sp and saving the results of each sub data package in a local database;

storing the size and the key of the first and the last record of each sub data package to a global database; and

subsequently identifying adjacent packages based on these small sub data packages by checking (“

not in parallel”

) the termination criterion for the loop i=i+1 being;

if key_pos1,xε

(key_pos1,y;

key<sub>pos max;

y</sub>) then continue else terminate, wherein pos1 illustrates the first position of a data package, posmax illustrates the last position of a data package, and x, y illustrates the number of data package, and if the conditional criterion is not accomplished for all combinations (x, y), meaning that the conditional inquiry is true, thereby comparing the key of the first record of each sub data package with the first and the last record of all the rest of sub data packages, assigning the sub data packages to a second parallel process for merging adjacent sub data packages so as to rebuild new data packages, wherein adjacent sub data packages are those data packages with keys of the first record which are closest together and have violated the termination criterion, and wherein by merging the sub data packages the original package size N is restored;

then storing the new data packages to a local database; and

subsequently assigning the new data packages again to the above mentioned first parallel process for reorganizing and sorting; and

thereafter aggregating the new data packages for key figures by using aggregation operations; and

following the aggregation, after each loop cycle splitting the aggregated data packages again into several smaller sub data packages and saving the results of each sub data package in a local database;

storing the size and the key of the first and the last record of each sub data package to a global database; and

subsequently identifying adjacent packages again based on these small packages by checking (“

not in parallel”

) the termination criterion for the loop i=i+1 for all combinations (x, y) anew, repeatedly executing the loop i=i+1 while the termination criterion for the loop is not accomplished, else after accomplishing the criterion, i.e. all the data packages are disjoint with regard to the granularity characteristics, terminating the loop i=i+1.

24. The computer program product of claim 22, wherein the program ultimately conducts an additional calculation step for enriching aggregated data packages, and wherein the data packages are subsequently saved to a global result database.

25. The computer program product of claim 21, further comprising the steps of:

enriching the prepared data packages in a parallel pre-processing step via secondary data source or data sources before the parallel aggregation starts; and

saving the results to a local database.

26. The computer program product of claim 24, further comprising the steps of:

enriching the aggregated data packages in a parallel post-processing step via secondary data source or data sources following the parallel aggregation; and

saving the results to a global result database.