System and method for managing throughput in the processing of query requests in a database system

• US 7,778,997 B1
• Filed: 06/11/2004
• Issued: 08/17/2010
• Est. Priority Date: 06/11/2004
• Status: Active Grant

• Current Assignee: LexisNexis Risk Data Management, Inc. (RELX PLC)
• Original Assignee: Seisint, Inc.
• Inventors: Halliday, Gavin C., Poulsen, Ole D., Bayliss, David A., Hicks, Nigel G., Cobbett-Smith, Jacob, Chapman, Richard K.
• Primary Examiner(s): Cottingham; John R
• Assistant Examiner(s): Reyes; Mariela D
• Application Number:
  US10/866,394
• Time in Patent Office: 2,258 Days
• Field of Search: 707/100, 707/9, 707/713, 707/718
• US Class Current: 707/713
• CPC Class Codes: G06F 16/2471 Distributed queries

Abstract

In certain embodiments, a database system for processing a query request comprises a first master node operable to receive a plurality of requests from a plurality of clients coupled to the database system. The first master node is capable of processing a first predetermined number of query requests substantially concurrently, each of the plurality of query requests comprising one or more parameters and being associated with at least one of a plurality of precompiled queries. The first master node is also operable to assign at least a first subset of the first predetermined number of query requests as being available to a first client, and to assign at least a second subset of the first predetermined number of query requests as being available to a second client.

Claims

1. A database system with a processor for processing a query request, the database system comprising a first master node operable to:

receive a plurality of requests from a plurality of clients coupled to the database system, the first master node being capable of processing a first predetermined number of query requests substantially concurrently, each of the plurality of query requests comprising one or more parameters used by at least one of a plurality of precompiled queries to resolve a query request, each precompiled query being a query that has been deployed on the database system in advance of a user executing such query on the database system;

assign, for processing, at least a first subset of the first predetermined number of query requests as being available to a first client;

assign, for processing, at least a second subset of the first predetermined number of query requests as being available to a second client; and

process the query request using a top level key to identify the location of a key part within the database system, the key part comprising stored data, an address associated with the stored data, and an identifier for identifying the location of additional data that satisfies at least a portion of the precompiled query, wherein the stored data is sorted according to combinations of the one or more parameters used by the precompiled query to resolve the query request.

2. The system of claim 1, wherein the first master node is further operable to receive a particular query request from the first client if a number of query requests concurrently being processed by the first master node for the first client is less than the first subset of the first predetermined number of query requests available to the first client.

3. The system of claim 2, wherein:

the system further comprises a second master node, the second master node being capable of processing a second predetermined number of query requests substantially concurrently, each of the plurality of query requests comprising one or more parameters and being associated with at least one of a plurality of precompiled queries; and

the second master node is operable to receive the particular query request from the first client, if a number of query requests concurrently being processed by the second master node is less than the second predetermined number of query requests.

4. The system of claim 2, further comprising a query receiving module operable to:

receive the particular query request from the first client; and

communicate the particular query request to the first master node if the number of query requests concurrently being processed by the first master node for the first client is less than the first subset of the first predetermined number of query requests available to the first client.

5. The system of claim 1, operable to ration CPU cycles for processing query requests received by the first master node based at least in part on the first client'"'"'s use of the first subset of query requests and the second client'"'"'s use of the second subset of query requests.

6. The system of claim 1, wherein the first master node comprises a particular number of threads each operable to receive a process at least one query request at a time, the first predetermined number of query requests being substantially equal to the particular number of threads.

7. The system of claim 1, wherein:

the system further comprises at least one slave node comprising a first incoming queue;

the first master node is operable to communicate a request associated with a particular query request received from the first client to the at least one slave node; and

the at least one slave node is operable to insert the request communicated by the first master node into the first incoming queue.

8. The system of claim 7, wherein:

the request communicated by the first master node is associated with a priority; and

the at least one slave node is operable to insert the request communicated by the master node into the first incoming queue according to the priority of the request.

9. The system of claim 7, wherein the first client is associated with a priority, the priority associated with the particular query request corresponding to the priority assigned to the first client.

10. The system of claim 7, wherein:

the request communicated by the master node is associated with a priority;

the slave node comprises a second incoming queue, the first incoming queue operable to store requests associated with a first priority value, the second incoming queue operable to store requests associated with a second priority value; and

the slave node is operable to insert the request communicated by the master node into either the first incoming queue or the second incoming queue according to the priority of the request.

11. A method for processing a query request in a database system, comprising:

receiving at a first master node of the database system a plurality of requests from a plurality of clients coupled to the database system, the first master node being capable of processing a first predetermined number of query requests substantially concurrently, each of the plurality of query requests comprising one or more parameters used by at least one of a plurality of precompiled queries to resolve a query request, each precompiled query being a query that has been deployed on the database system in advance of a user executing such query on the database system;

assigning, for processing, at least a first subset of the first predetermined number of query requests as being available to a first client;

assigning, for processing, at least a second subset of the first predetermined number of query requests as being available to a second client; and

processing the query request using a top level key to identify the location of a key part within the database system, the key part comprising stored data, an address associated with the stored data, and an identifier for identifying the location of additional data that satisfies at least a portion of the precompiled query, wherein the stored data is sorted according to combinations of the one or more parameters used by the precompiled query to resolve the query request.

12. The method of claim 11, further comprising receiving at the first master node a particular query request from the first client if a number of query requests concurrently being processed by the first master node for the first client is less than the first subset of the first predetermined number of query requests available to the first client.

13. The method of claim 12, wherein:

the system further comprises a second master node, the second master node being capable of processing a second predetermined number of query requests substantially concurrently, each of the plurality of query requests comprising one or more parameters and being associated with at least one of a plurality of precompiled queries; and

the method further comprises receiving at the second master node the particular query request from the first client, if a number of query requests concurrently being processed by the second master node is less than the second predetermined number of query requests.

14. The method of claim 12, further comprising communicating the particular query request from a query receiving module to the first master node if the number of query requests concurrently being processed by the first master node for the first client is less than the first subset of the first predetermined number of query requests available to the first client.

15. The system of claim 11, operable to ration CPU cycles for processing query requests received by the first master node based at least in part on the first client'"'"'s use of the first subset of query requests and the second client'"'"'s use of the second subset of query requests.

16. The method of claim 11, wherein the first master node comprises a particular number of threads each operable to receive and process at least one query request at a time, the first predetermined number of query requests being substantially equal to the particular number of threads.

17. The method of claim 11, wherein:

the method further comprises communicating from the first master node a request associated with a particular query request received from the first client to at least one slave node of the database system; and

the at least one slave node is operable to insert the request communicated by the first master node into a first incoming queue of the at least one slave node.

18. The method of claim 17, further comprising:

associating the request communicated by the first master node with a priority; and

inserting the request communicated by the master node into the first incoming queue according to the priority of the request.

19. The method of claim 17, comprising associating the first client with a priority, the priority associated with the particular query request corresponding to the priority assigned to the first client.

20. The method of claim 17, further comprising:

associating the request communicated by the master node with a priority; and

inserting the request communicated by the master node into either the first incoming queue of the slave node or a second incoming queue of the slave node according to the priority of the request, the first incoming queue operable to store requests associated with a first priority value, the second incoming queue operable to store requests associated with a second priority value.

21. A computer-readable storage medium, storing program instructions for performing, when executed by a processor, a method for processing a query request in a database system, comprising:

receiving at a first master node of the database system a plurality of requests from a plurality of clients coupled to the database system, the first master node being capable of processing a first predetermined number of query requests substantially concurrently, each of the plurality of query requests comprising one or more parameters used by at least one of a plurality of precompiled queries to resolve a query request, each precompiled query being a query that has been deployed on the database system in advance of a user executing such query on the database system;

assigning, for processing, at least a first subset of the first predetermined number of query requests as being available to a first client;

assigning, for processing, at least a second subset of the first predetermined number of query requests as being available to a second client; and

processing the query request using a top level key to identify the location of a key part within the database system, the key part comprising stored data, an address associated with the stored data, and an identifier for identifying the location of additional data that satisfies at least a portion of the precompiled query, wherein the stored data is sorted according to combinations of the one or more parameters used by the precompiled query to resolve the query request.

22. The computer-readable storage medium of claim 21, further operable to receive at the first master node a particular query request from the first client if a number of query requests concurrently being processed by the first master node for the first client is less than the first subset of the first predetermined number of query requests available to the first client.