Query scheduling in a parallel-processing database system

• US 7,185,003 B2
• Filed: 11/14/2002
• Issued: 02/27/2007
• Est. Priority Date: 11/14/2002
• Status: Active Grant

• Current Assignee: LexisNexis Risk Data Management, Inc. (RELX PLC)
• Original Assignee: Seisint, Inc.
• Inventors: Bayliss, David, Smith, Jake, Hicks, Nigel, Halliday, Gavin, Poulsen, Ole, Chapman, Richard
• Primary Examiner(s): HWANG, JOON H
• Application Number:
  US10/293,489
• Publication Number:
  US 20040098374A1
• Time in Patent Office: 1,566 Days
• Field of Search: 707/3, 707/10, 718/102, 709201-203
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system and method for scheduling database operations to one or more databases in a parallel-processing database system are described herein. After a query server generates a dynamic-link library (DLL) or other executable representative of one or more database operations to a database, the query server notifies a scheduling services module of the generation of the DLL and submits the DLL to a query agent. The query agent notifies the scheduling services module of its receipt of the DLL. Based on any of a variety of considerations, the scheduling services module schedules a time of execution for the DLL by one or more processing matrices that store the database. At the scheduled time, the scheduling services module directs the query agent to submit the DLL to the indicated processing matrices. The scheduling services module also can be adapted to monitor the execution of previously submitted DLLs by one or more processing matrices and adjust the scheduled times of execution for subsequent DLLs accordingly.

Claims

1. A system for scheduling database operations on at least one database, the system comprising:

a first global-results processing matrix having a plurality of interconnected processing nodes operating in parallel and being adapted to execute an executable on the at least one database, the executable being representative of a compiled query having at least one database operation;

a first query agent connected to the first global-results processing matrix and being adapted to manage the execution of the executable by the first global-results processing matrix; and

a scheduling services module connected to the query agent and the first global-results processing matrix and being adapted to;

schedule a time for the execution of the executable by the first global-results processing matrix;

adjust the scheduled time of the execution of the executable based on information received from the global-results processing matrix or the first query agent; and

direct the first query agent to submit the executable to the first global-results processing matrix for the execution at the adjusted scheduled time of the execution.

2. The system of claim 1, wherein the scheduling services module is further adapted to adjust the scheduled time of the execution for the executable based in part on a difference between an expected execution time and an actual execution time of at least one executable previously executed by the first global-results processing matrix.

3. The system of claim 2, wherein the first query agent is further adapted to provide an indication of a progress of an execution of an executable by the first global-results processing matrix to the scheduling services module.

4. The system of claim 1, further comprising:

a second global-results processing matrix having a plurality of interconnected processing nodes operating in parallel and being adapted to execute a second executable on the at least one database, the second executable being representative of a compiled query having at least one database operation;

a second query agent connected to the second global-results processing matrix and the scheduling services module and being adapted to manage the execution of the second executable by the second global-results processing matrix; and

wherein the scheduling services module is further adapted to;

schedule a time for the execution of the second executable by the second global-results processing matrix; and

direct the second query agent to submit the second executable to the second global-results processing matrix for the execution at the scheduled time of the execution of the second executable.

5. The system of claim 4, wherein the scheduling services module is further adapted to schedule executions of executables at the first global-results processing matrix and the second global-results processing matrix to minimize execution times for the executables.

6. The system of claim 4, wherein the scheduling services module is further adapted to schedule executions of executables at the first global-results processing matrix and the second global-results processing matrix to maximize utilization of the first global-results processing matrix and the second global-results processing matrix.

7. The system of claim 4, wherein the scheduling services module is further adapted to schedule executions of executables at the first global-results processing matrix and the second global-results processing matrix to minimize delays in execution of the executables by the first global-results processing matrix and the second global-results processing matrix.

8. The system of claim 1, further comprising:

a general-purpose query processing matrix having a plurality of interconnected processing nodes operating in parallel and being adapted to execute a third executable on the at least one database, the third executable being representative of a compiled query having at least one database operation;

a second query agent connected to the general-purpose query processing matrix and the scheduling services module and being adapted to manage the execution of the third executable by the general-purpose query processing matrix; and

wherein the scheduling services module is further adapted to;

schedule a time for the execution of the third executable by the general-purpose query processing matrix; and

direct the second query agent to submit the third executable to the general-purpose query processing matrix for the execution at the scheduled time of the execution of the third executable.

9. The system of claim 8, wherein the scheduling services module is further adapted to schedule executions of executables at the first global-results processing matrix and the general-purpose query processing matrix to minimize execution times for the executables.

10. The system of claim 8, wherein the scheduling services module is further adapted to schedule executions of executables at the first global-results processing matrix and general-purpose query processing matrix to maximize utilization of the first global-results processing matrix and the general-purpose query processing matrix.

11. The system of claim 8, wherein the scheduling services module is further adapted to schedule executions of executables at the first global-results processing matrix and the second global-results processing matrix to minimize delays in execution of the executables by the first global-results processing matrix and the general-purpose query processing matrix.

12. In a parallel processing based database management system, a method for scheduling execution of compiled executables representing queries having at least one database operation, the method comprising:

scheduling a time for execution of an executable by a first global-results processing matrix, the first global-results processing matrix having a plurality of interconnected processing nodes operating in parallel and being adapted to execute the executable on at least one database;

adjusting the scheduled time of the execution of the executable based on information received from the first global-results processing matrix or a first query agent, the first query agent connected to the first global-results processing matrix and being adapted to manage the execution of the executable by the first global- results processing matrix; and

submitting, at the adjusted scheduled time, the executable to the first global-results processing matrix for the execution via the first query agent.

13. The method of claim 12, further comprising adjusting the scheduled time of the execution of the executable by the first global-results processing matrix based in part on a difference between an expected execution time and an actual execution time of at least one executable previously executed by the first global-results processing matrix.

14. The method of claim 12, further comprising:

scheduling a time for execution of a second executable by a second global-results processing matrix; and

submitting, at the scheduled time for the second executable, the second executable to the second global-results processing matrix for execution.

15. The method of claim 14, wherein the scheduling of the executions of the executables at the first global-results processing matrix and the second global-results processing matrix is determined to minimize execution times for the executables.

16. The method of claim 14, wherein the scheduling of the executions of the executables at the first global-results processing matrix and the second global-results processing matrix is determined to maximize utilization of the first global-results processing matrix and the second global-results processing matrix.

17. The method of claim 14, wherein the scheduling of the executions of the executables at the first global-results processing matrix and the second global-results processing matrix is determined to minimize delays in execution of the executables by the first global-results processing matrix and the second global-results processing matrix.

18. The method of claim 12, further comprising:

scheduling a time for execution of a third executable by a general-purpose query processing matrix; and

submitting, at the scheduled time for the third executable, the third executable to the general-purpose processing matrix for execution.

19. The method of claim 18, wherein the scheduling of the executions of the executables at the first global-results processing matrix and the general-purpose query processing matrix is determined to minimize execution times for the executables.

20. The method of claim 18, wherein the scheduling of the executions of the executables at the first global-results processing matrix and the general-purpose query processing matrix is determined to maximize utilization of the first global-results processing matrix and the general-purpose query processing matrix.

21. The method of claim 18, wherein the scheduling of the executions of the executables at the first global-results processing matrix and the general-purpose query processing matrix is determined to minimize delays in execution of the executables by the first global-results processing matrix and the general-purpose query processing matrix.

22. The method of claim 12, further comprising:

scheduling a time for execution of a fourth executable by an index-based query processing matrix; and

submitting, at the scheduled time for the fourth executable, the fourth executable to the index-based query processing matrix for execution.

23. The method of claim 21, wherein the scheduling of the executions of the executables at the first global-results processing matrix and an the index-based query processing matrix is determined to minimize execution times for the executables.

24. The method of claim 22, wherein the scheduling of the executions of the executables at the first global-results processing matrix and an index-based query processing matrix is determined to maximize utilization of the first global-results processing matrix and the index-based query processing matrix.

25. The method of claim 22, wherein the scheduling of the executions of the executables at the first global-results processing matrix and an index-based query processing matrix is determined to minimize delays in execution of the executables by the first global-results processing matrix and the index-based query processing matrix.