Network interface for distributed intelligence database system

• US 7,272,605 B1
• Filed: 05/13/2002
• Issued: 09/18/2007
• Est. Priority Date: 05/13/2002
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: Netezza (International Business Machines Corporation)
• Inventors: Zane, Barry M., Hinshaw, Foster D., Utter, David A., Almy, Steven T.
• Primary Examiner(s): PARDO, THUY N
• Application Number:
  US10/145,564
• Time in Patent Office: 1,954 Days
• Field of Search: 707/8, 707/10, 707/102, 707/3, 707/200, 707/204, 707/206, 705/5, 709/203, 709/225, 709/226, 709/227, 235/380, 711/114, 714/5
• US Class Current: 707/688
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: Y10S 707/922 Communications
• CPC Class Codes: Y10S 707/955 Object-oriented
• CPC Class Codes: Y10S 707/959 Network
• CPC Class Codes: Y10S 707/966 Distributed
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A database appliance having multiple internetworked data storage units and central database processing units. The central database processors and storage units communicate as network nodes. Execution of a distributed database application is coordinated by a communication process such that data blocks are passed by using data block reference information. Operations executing within the same context as the communication process may include join, sort, aggregate, restrict, reject, expression evaluation, statistical analysis or other database operations.

Claims

1. A distributed data processing system for executing database operations comprising:

(a) two or more data storage processors for coordinating and processing the storage of database records;

(b) at least one central database processor for processing database operations;

(c) a network infrastructure providing for network connection of the data storage processors and at least one central database processor as network nodes, to support communication of data among network nodes, the network infrastructure using a communication protocol;

(d) a communication process for communicating data blocks in one or more messages communicated among network nodes;

(e) an application process for executing database operations, with portions of the application process executing on the data storage processors, and portions of the application process executing on at least one central database processor; and

(f) the application process and the communication process passing data blocks via data block reference information.

2. The system as in claim 1 wherein a data storage processor also serves as a central database processor.

3. The system as in claim 1 wherein the data block reference information is a list of data blocks.

4. The system as in claim 1 wherein the communication process uses a layered communication protocol and a network layer in the communication process communicates with the application process via data block reference information.

5. The system as in claim 1 wherein the data blocks contain an integral number of database objects resulting from a database operation.

6. The system as in claim 4 wherein layers of the communication protocol write layer-specific information to a header block and/or a trailer block prior to passing the data block reference information to another layer.

7. The system as in claim 3 wherein the data storage area is in a memory area allocated to be shared by a sending application process and an associated communication process.

8. The system as in claim 7 wherein the data storage area containing one or more data blocks in a sending application process is not released until a receiving application process at a destination node acknowledges receipt.

9. The system as in claim 1 wherein a data storage area associated with a receiving process maintains state information for the communication process in the same table as state information for the application process.

10. The system as in claim 1 wherein a data storage area associated with a receiving process maintains state information for the communication process in a different table from the table in which state information for the application process is maintained.

11. The system as in claim 1 wherein the messages are communicated in multiple packets and the packets include packet sequence information used by a receiving process to determine if packets are received out of order.

12. A distributed data processing system for executing database operations comprising:

(a) two or more data storage processors for coordinating and processing the storage of database records;

(b) at least one central database processor for processing database operations;

(c) a network infrastructure providing network connection of the data storage processors and at least one central database processor as network nodes, to support communication of data among network nodes, the network infrastructure using a communication protocol;

(d) a communication process responsible for communicating data blocks in one or more messages communicated among network nodes,

(e) an application process for executing database operations, with portions of the application process executing on the data storage processors, and portions of the application process executing on at least one central database processor; and

(f) the communication process having access to information indicating application specific payload data types in the communicated messages and making processing decisions based thereon.

13. The system as in claim 12 wherein a data storage processor also serves as a central database processor.

14. The system as in claim 12 wherein the processing decisions are decisions with respect to routing messages within the network infrastructure.

15. The system as in claim 12 wherein a predefined payload data type indicates an integral number of database records are contained in each communication packet.

16. The system as in claim 15 wherein the predefined payload data type indicates that a streaming protocol is to be used for communicating messages.

17. The system as in claim 12 wherein a predefined payload data type indicates that a group of related packets contains an integral number of database records.

18. The system as in claim 12 wherein a predefined payload data type indicates that messages contain groups of related database objects.

19. The system as in claim 12 wherein a portion of the application process executes on a network interface component.

20. A distributed data processing system for executing database operations comprising:

(a) two or more data storage processors for coordinating and processing the storage of database records;

(b) at least one central database processor for processing database operations;

(c) a network infrastructure providing for network connection of the data storage processors and at least one central database processor as network nodes, to support communication of data among network nodes, the network infrastructure using a communication protocol;

(d) a communication process responsible for communicating data blocks in one or more messages communicated among network nodes;

(e) an application process for executing database operations, with portions of the application process executing on the data storage processors, and portions of the application process executing on at least one central database processor; and

(f) the communication process formulating messages as at least one packet, such that each packet contains an integral number of database objects.

21. The system as in claim 20 wherein a data storage processor also serves as a central database processor.

22. The system as in claim 20 wherein an application process in a receiver begins processing received packers prior to receipt of a complete message.

23. The system as in claim 20 wherein an application process in a sender queues packets prior to their transmission, and the application process begins sending queued packets prior to availability of a complete message.

24. The system as in claim 20 wherein a communication process in a receiver processes receives packets and extracts database records independent of the application process.

25. The system as in claim 20 wherein a communication process in a sender processes assembles database records into packets prior to the availability of a complete set of database records.

26. The system as in claim 24 wherein the communication process in the receiver processes received records according to database application process operations.

27. The system as in claim 25 wherein the communication process in the sender filters records according to database application process operations.

28. The system as in claim 20 wherein the application process operations are database operations.

29. The system as in claim 28 wherein the database operations are selected from a group consisting of join, sort, aggregate, restrict, reject, expression evaluation or statistical analysis.

30. The system as in claim 20 wherein the portion of the application process executing on a central database processor executes in a central processing unit.

31. The system as in claim 20 wherein the portion of the application process executes in a network interface component associated with the central database processor.

32. The system as in claim 27 wherein the database operation results from the communication process are stored in a common memory accessible to the application process.

33. A distributed data processing system for executing database operations comprising:

(a) two or more data storage processors for coordinating the storage of database records;

(b) at least one central database processor for processing database operations;

(c) a network infrastructure providing for network connection of the data storage processors and at least one central database processor as network nodes, to support communication of data among network nodes, the network infrastructure using a communication protocol;

(d) a communication process principally responsible for communicating data blocks in one or more messages communicated among network nodes;

(e) an application process for executing database operations, with portions of the application process executing on the data storage processors, and portions of the application process executing on at least one central database processor;

(f) the communication process making use of a communication protocol that does not provide for reliable message delivery, and

(g) the application process being augmented with a reliable delivery paradigm for processing messages.

34. The system as in claim 33 wherein a data storage processor also serves as a central database processor.

35. The system as in claim 33 wherein a portion of the application process executes on a network interface component.

36. The system as in claim 33 wherein the reliable delivery paradigm is an end-to-end acknowledgment.

37. The system as in claim 36 wherein the application process sends the acknowledgement as part of another message.

38. The system as in claim 36 wherein the application layer sends the acknowledgement as a separate message.

39. The system as in claim 38 wherein the separate message is not sent until after expiration of a predetermined period of time without having sent other routine messages.

40. The system as in claim 33 wherein the reliable delivery paradigm uses messages sent as packets with packet sequence numbers.

41. The system as in claim 33 wherein the communication protocol is User Datagram Protocol (UDP).