Grid computing system alongside a distributed database architecture

• US 8,271,537 B2
• Filed: 11/15/2010
• Issued: 09/18/2012
• Est. Priority Date: 11/15/2010
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Schabenberger, Oliver, Krueger, Steve
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US12/946,079
• Publication Number:
  US 20120124100A1
• Time in Patent Office: 673 Days
• Field of Search: 707/791, 707/792, 707/797, 707/802
• US Class Current: 707/802
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Systems and methods are provided for a grid computing system that performs analytical calculations on data stored in a distributed database system. A grid-enabled software component at a control node is configured to invoke database management software (DBMS) at the control node to cause the DBMS at a plurality of the worker nodes to make available data to the grid-enabled software component local to its node; instruct the grid-enabled software components at the plurality of worker nodes to perform an analytical calculation on the received data and to send the results of the data analysis to the grid-enabled software component at the control node; and assemble the results of the data analysis performed by the grid-enabled software components at the plurality of worker nodes.

Claims

1. A grid computing system for performing analytical calculations on data stored in a distributed database system, the grid computing system comprising:

a plurality of grid computing nodes including a control node and a plurality of worker nodes, each node comprising;

a processor for executing database management software (DBMS) and for executing a grid-enabled software component,

local buffer memory accessible by the grid-enabled software component, and

local storage media accessible to the DBMS for storing data;

wherein the grid-enabled software component is configured to perform the analytical calculations on a portion of the data stored by the database management system;

the grid-enabled software component at the control node being configured to;

invoke the DBMS at the control node to cause the DBMS at a plurality of the worker nodes to make available a portion of the data to the grid-enabled software component local to its node;

instruct the grid-enabled software components at the plurality of worker nodes to perform an analytical calculation on the received data and to send the results of the data analysis to the grid-enabled software component at the control node; and

assemble the results of the data analysis performed by the grid-enabled software components at the plurality of worker nodes.

2. The system according to claim 1 wherein the DBMS at a plurality of worker nodes are configured to make the data available to their local grid-enabled software component by saving the data to shared memory locally accessible by both the DBMS and the grid-enabled software component.

3. The system according to claim 1 wherein the DBMS at the plurality of worker nodes are configured to make the data available to their local grid-enabled software component by communicating over a socket connection.

4. The system according to claim 3 wherein the grid-enabled software components at the plurality of worker nodes are configured to open socket connections using a common port address.

5. The system according to claim 4 wherein the grid-enabled software component at the control node is configured to provide the common port address to the DBMS local to its node and the DBMS at the control node is configured to provide the common port address to the DBMS at the plurality of worker nodes.

6. The system according to claim 1 wherein the grid-enabled software components at the plurality of worker nodes are configured to store the received data and perform the data analysis using their local buffer memory.

7. The system according to claim 1 wherein the plurality of worker nodes are configured to transmit the results from the data analysis to their local DBMS for storage in the distributed database.

8. The system according to claim 1 wherein the results of the analysis received from the plurality of worker node grid-enabled software components are received via a network.

9. The system according to claim 8 wherein a plurality of the worker node grid-enabled software components transmit their results using MPI over the network.

10. The system according to claim 1 wherein the results assembled by the control node grid-enabled software component are stored in the distributed database system.

11. The system according to claim 1 wherein the results assembled by the control node grid-enabled software component are transmitted to a grid-enabled software component at a client node.

12. The system according to claim 1 wherein upon the detection of a failure at one of the worker nodes:

the DBMS at the control node is configured to cause the DBMS at one of the remaining worker nodes to redistribute the portion of the data previously distributed to the grid-enabled software component at the failed node to the grid-enabled software component local to that node;

the grid-enabled software component receiving the redistributed data is configured to perform an analytical calculation on the redistributed data and provide the results to the grid-enabled software component at the control node; and

the grid-enabled software component at the control node is configured to assemble the results received from the grid-enabled software components at the remaining worker nodes.

13. The system according to claim 1, further comprising a grid-enabled software component at a client node:

the grid-enabled software component at the client mode being configured with a first executing mode and a second executing mode;

the client node grid-enabled software component in the first executing mode being configured to perform local data analysis for provision of the results to the client software;

the client node grid-enabled software component in the second executing mode being configured to invoke the grid-enabled software component at the control node to perform the data analysis in a grid-based computing environment; and

the grid-enabled software component at the client node being configured to operate in the first mode or the second mode based on an indication provided by the client software.

14. The system according to claim 13 wherein the client node grid-enabled software component is configured to receive ad hoc questions from a client application and to provide responses to the ad hoc questions by providing the results assembled by the control node grid-enabled software component to the client application.

15. The system according to claim 1, further comprising a client-node grid-enabled software component that is configured to receive ad hoc questions from a client application and to provide responses to the ad hoc questions by providing the results assembled by the control node grid-enabled software component to the client application.

16. The system of claim 1, wherein the grid-enabled software component at the control node is configured to assemble the results by aggregating and concatenating the results of the analysis performed at the worker node data processors.

17. The system of claim 1, wherein the grid-enabled software component at a plurality of the grid computing nodes is implemented in a virtual machine environment at each node.

18. The system of claim 1, wherein the database management software at a plurality of the grid computing nodes is implemented in a virtual machine environment at each node.

19. The system of claim 18, wherein the grid-enabled software component at the plurality of the grid computing nodes is implemented in a separate virtual machine environment at each node.

20. The system of claim 18, wherein the grid-enabled software component at the plurality of the grid computing nodes is implemented in a common virtual machine environment with the database management software at each node.

21. A method in a grid computing system for performing analytical calculations on data stored in a distributed database system, the method comprising:

providing a plurality of grid computing nodes including a control node and a plurality of worker nodes, each node comprising a processor for executing database management software (DBMS) and for executing a grid-enabled software component, local buffer memory accessible by the grid-enabled software component, and local storage media accessible to the DBMS for storing data;

invoking the DBMS at the control node to cause the DBMS at a plurality of the worker nodes to make available a portion of the data to the grid-enabled software component local to its node;

instructing the grid-enabled software components at the plurality of worker nodes to perform an analytical calculation on the received data and to send the results of the data analysis to the grid-enabled software component at the control node; and

assembling the results of the data analysis performed by the grid-enabled software components at the plurality of worker nodes.

22. The method according to claim 21 wherein the DBMS at a plurality of worker nodes are configured to make the data available to their local grid-enabled software component by saving the data to shared memory locally accessible by both the DBMS and the grid-enabled software component.

23. The method according to claim 21 wherein the DBMS at the plurality of worker nodes are configured to make the data available to their local grid-enabled software component by communicating over a socket connection.

24. The method according to claim 23 wherein the grid-enabled software components at the plurality of worker nodes are configured to open socket connections using a common port address.

25. The method according to claim 24 wherein the grid-enabled software component at the control node is configured to provide the common port address to the DBMS local to its node and the DBMS at the control node is configured to provide the common port address to the DBMS at the plurality of worker nodes.

26. The method according to claim 21 wherein the grid-enabled software components at the plurality of worker nodes are configured to store the received data and perform the data analysis using their local buffer memory.

27. The method according to claim 21 wherein the plurality of worker nodes are configured to transmit the results from the data analysis to their local DBMS for storage in the distributed database.

28. The method according to claim 21 wherein the results of the analysis received from the plurality of worker node grid-enabled software components are received via a network.

29. The method according to claim 28 wherein a plurality of the worker node grid-enabled software components transmit their results using MPI over the network.

30. The method according to claim 21 further comprising storing the results assembled by the control node grid-enabled software component in the distributed database system.

31. The method according to claim 21 further comprising transmitting the results assembled by the control node grid-enabled software component to a grid-enabled software component at a client node.

32. The method according to claim 21 wherein upon the detection of a failure at one of the worker nodes:

the DBMS at the control node is configured to cause the DBMS at one of the remaining worker nodes to redistribute the portion of the data previously distributed to the grid-enabled software component at the failed node to the grid-enabled software component local to that node;

the grid-enabled software component receiving the redistributed data is configured to perform an analytical calculation on the redistributed data and provide the results to the grid-enabled software component at the control node; and

the grid-enabled software component at the control node is configured to assemble the results received from the grid-enabled software components at the remaining worker nodes.

33. The method according to claim 21, further comprising the step of providing a grid-enabled software component at a client node wherein:

the grid-enabled software component at the client mode is configured with a first executing mode and a second executing mode;

the client node grid-enabled software component in the first executing mode is configured to perform local data analysis for provision of the results to the client software;

the client node grid-enabled software component in the second executing mode is configured to invoke the grid-enabled software component at the control node to perform the data analysis in a grid-based computing environment; and

the grid-enabled software component at the client node is configured to operate in the first mode or the second mode based on an indication provided by the client software.

34. The method according to claim 33 wherein the client node grid-enabled software component is configured to receive ad hoc questions from a client application and to provide responses to the ad hoc questions by providing the results assembled by the control node grid-enabled software component to the client application.

35. The method according to claim 21, further comprising providing a client-node grid-enabled software component that is configured to receive ad hoc questions from a client application and to provide responses to the ad hoc questions by providing the results assembled by the control node grid-enabled software component to the client application.

36. The method of claim 21, wherein the assembling step comprises aggregating and concatenating the results of the analysis performed at the worker node data processors.