System and architecture for enterprise-scale, parallel data mining

• US 20070174290A1
• Filed: 01/19/2006
• Published: 07/26/2007
• Est. Priority Date: 01/19/2006
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Natarajan, Ramesh, Narang, Inderpal, Sioh, Radu
• Application Number:
  US11/335,371
• Publication Number:
  US 20070174290A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A grid-based approach for enterprise-scale data mining that leverages database technology for I/O parallelism and on-demand compute servers for compute parallelism in the statistical computations is described. By enterprise-scale, we mean the highly-automated use of data mining in vertical business applications, where the data is stored on one or more relational database systems, and where a distributed architecture comprising of high-performance compute servers or a network of low-cost, commodity processors, is used to improve application performance, provide better quality data mining models, and for overall workload management. The approach relies on an algorithmic decomposition of the data mining kernel on the data and compute grids, which provides a simple way to exploit the parallelism on the respective grids, while minimizing the data transfer between them. The overall approach is compatible with existing standards for data mining task specification and results reporting in databases, and hence applications using these standards-based interfaces do not require any modification to realize the benefits of this grid-based approach.

Claims

1. A system comprising:

(i) a data grid comprising a collection of disparate data repositories;

(ii) a compute grid comprising a collection of disparate compute resources; and

(iii) means for combining the data grid and the compute grid so that in operation they are suitable for processing business applications of at least one of data modeling and model scoring.

2. A system according to claim 1, comprising means so that the data grid and the compute grid include algorithmic decomposition of a data mining kernel on the data and compute grids thereby enabling parallelism on the respective grids, while minimizing data transfer between the respective grids.

3. A system according to claim 1, wherein the data grid comprises a parameterized task estimator for enabling run time estimation and task-resource matching algorithms.

4. A system according to claim 1, wherein the compute grid comprises a set of scheduling algorithms guided by data driven requirements for enabling resource matching and utilization of the compute grid.

5. A system according to claim 1, wherein the parallel compute grid comprises numerous preloaded models in the individual node memories for scalable interactive scoring, thereby avoiding the overhead of keeping the models in the limited memory of the data server or reading from the data server disk which limits the required fast interactive response.