PERFORMING DATA MINING OPERATIONS WITHIN A COLUMNAR DATABASE MANAGEMENT SYSTEM

• US 20140372482A1
• Filed: 06/12/2014
• Published: 12/18/2014
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Open Text Holdings, Inc. (Open Text Corporation)
• Original Assignee: Actuate Corporation (Open Text Corporation)
• Inventors: Martn, Carles Bays, Poyatos, Jess Malo, Sierra, Marc Rodrguez, Prez, Alejandro Sualdea
• Granted Patent: US 9,798,783 B2
• Time in Patent Office: Days
• US Class Current: 707/776
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases

Abstract

Data mining operations are performed within a columnar database management system. The columnar database management system stores input sets of data for a data mining operation. An input set of data is represented as a column of data in the columnar database management system. The columnar database management system stores instructions to perform one or more data mining operations for processing the input sets of data. The columnar database management system receives requests for performing data mining operations and performs the processing of the data mining operation within the columnar database management system. As a result, the processing of data mining operations is performed without requiring multiple data transfers between an application implementing the data mining operations and the columnar database management system.

Claims

1. A computer-implemented method for performing data mining operations using a columnar database management system, the method comprising:

storing one or more input sets of data for a data mining operation within a columnar database management system, an input set represented as a column of data;

storing, in the columnar database management system, instructions to perform one or more data mining operations for processing the input sets of data;

receiving, from a client, a request to perform a data mining operation on the one or more input sets of data; and

responsive to receiving the request, obtaining by the columnar database management system, a set of data mining results, the obtaining comprising;

parsing the data mining request within the columnar database management system,

generating a data mining processing plan comprising various steps for processing the requested data mining operation, and

executing the data mining processing plan to obtain the results of the requested data mining operation; and

sending the set of data mining results obtained to a client device sending the request.

2. The computer-implemented method of claim 1, wherein the data mining operation comprises one of:

clustering, classification, regression analysis, summarization, association mining, or anomaly detection.

3. The computer-implemented method of claim 1, wherein the data mining operation is a first data mining operation and the data mining processing plan comprises at least a step executing a second data mining operation, wherein an output of the second data mining operation is provided as input to a subsequent step of the data mining processing plan, the output of the second data mining operation transferred to a subsequent step of the data mining processing plan, the transfer performed within the columnar database management system.

4. The computer-implemented method of claim 3, wherein the first data mining operation is a decision tree operation and the second data mining operation is a cross tabulation operation.

5. The computer-implemented method of claim 1, further comprising:

estimating an amount of memory needed for the data mining operation based on an amount of memory used by a previous execution of the corresponding data mining operation; and

using the estimated amount of memory for memory pre-allocation for the data mining operation.

6. The computer-implemented method of claim 1, further comprising:

estimating a duration of execution of the data mining operation based on a duration of execution of a previous execution of the corresponding data mining operation; and

using the estimated duration of execution for optimizing the execution of the data mining operation.

7. The computer-implemented method of claim 1, wherein the data mining processing plan includes a first step processing a first set of input sets and a second step processing a second set of input sets, the method further comprising:

determining an order of execution of the steps of the data mining processing plan such that the second step is executed after the first step responsive to the first set of input sets being included in the second set of input sets.

8. The computer-implemented method of claim 1, wherein the data mining operations comprise a cross tabulation operation and processing the cross tabulation operation comprises:

for each row combination of result, wherein the row combination has one or more attribute values of input columns;

identifying data of input sets having the attribute values;

determining an intersection set of the identified data; and

determining the output value for the combination of result based on the intersection set.

9. The computer-implemented method of claim 1, wherein the data mining operation comprises generation of Venn diagram.

10. The computer-implemented method of claim 1, wherein the data mining operations comprise a decision tree operation to classify a set of data items into various groups.

11. The computer-implemented method of claim 1, wherein the data mining operations comprise one or more of association rule algorithms, clustering algorithms, or time series algorithm.

12. A non-transitory computer readable storage medium storing instructions for:

storing one or more input sets of data for a data mining operation within a columnar database management system, an input set represented as a column of data;

storing, in the columnar database management system, instructions to perform one or more data mining operations for processing the input sets of data;

receiving, from a client, a request to perform a data mining operation on the one or more input sets of data; and

responsive to receiving the request, obtaining by the columnar database management system, a set of data mining results, the obtaining comprising;

parsing the data mining request within the columnar database management system,

generating a data mining processing plan comprising various steps for processing the requested data mining operation, and

executing the data mining processing plan to obtain the results of the requested data mining operation; and

sending the set of data mining results obtained to a client device sending the request.

13. The non-transitory computer readable storage medium of claim 12, wherein the data mining operation is a first data mining operation and the data mining processing plan comprises at least a step executing a second data mining operation, wherein an output of the second data mining operation is provided as input to a subsequent step of the data mining processing plan, the output of the second data mining operation transferred to a subsequent step of the data mining processing plan, the transfer performed within the columnar database management system.

14. The non-transitory computer readable storage medium of claim 12, further comprising instructions for:

estimating an amount of memory needed for the data mining operation based on an amount of memory used by a previous execution of the corresponding data mining operation; and

using the estimated amount of memory for memory pre-allocation for the data mining operation.

15. The non-transitory computer readable storage medium of claim 12, further comprising instructions for:

estimating a duration of execution of the data mining operation based on a duration of execution of a previous execution of the corresponding data mining operation; and

using the estimated duration of execution for optimizing the execution of the data mining operation.

16. The non-transitory computer readable storage medium of claim 12, wherein the data mining processing plan includes a first step processing a first set of input sets and a second step processing a second set of input sets, further comprising instructions for:

determining an order of execution of the steps of the data mining processing plan such that the second step is executed after the first step responsive to the first set of input sets being included in the second set of input sets.

17. The non-transitory computer readable storage medium of claim 12, wherein the data mining operation is a cross tabulation operation and processing the cross tabulation operation comprises:

for each row combination of result, wherein the row combination has one or more attribute values of input columns;

identifying data of input sets having the attribute values;

determining an intersection set of the identified data; and

determining the output value for the combination of result based on the intersection set.

18. A computer-implemented system comprising:

a computer processor; and

a non-transitory computer-readable storage medium storing instructions for;

storing one or more input sets of data for a data mining operation within a columnar database management system, an input set represented as a column of data;

storing, in the columnar database management system, instructions to perform one or more data mining operations for processing the input sets of data;

receiving, from a client, a request to perform a data mining operation on the one or more input sets of data; and

responsive to receiving the request, obtaining by the columnar database management system, a set of data mining results, the obtaining comprising;

parsing the data mining request within the columnar database management system,

generating a data mining processing plan comprising various steps for processing the requested data mining operation, and

executing the data mining processing plan to obtain the results of the requested data mining operation; and

sending the set of data mining results obtained to a client device sending the request.

19. The non-transitory computer readable storage medium of claim 18, wherein the data mining operation is a first data mining operation and the data mining processing plan comprises at least a step executing a second data mining operation, wherein an output of the second data mining operation is provided as input to a subsequent step of the data mining processing plan, the output of the second data mining operation transferred to a subsequent step of the data mining processing plan, the transfer performed within the columnar database management system.

20. The non-transitory computer readable storage medium of claim 18, wherein the non-transitory computer-readable storage medium further stores instructions for:

estimating an amount of memory needed for the data mining operation based on an amount of memory used by a previous execution of the corresponding data mining operation; and

using the estimated amount of memory for memory pre-allocation for the data mining operation.