Continuously available database server having multiple groups of nodes, each group maintaining a database copy with fragments stored on multiple nodes

• US 5,423,037 A
• Filed: 11/08/1994
• Issued: 06/06/1995
• Est. Priority Date: 03/17/1992
• Status: Expired due to Term

• Current Assignee: Sun Microsystems Incorporated (Oracle Corporation)
• Original Assignee: Teleserve Transaction Technology As
• Inventors: Hvasshovd, Svein-Olaf
• Primary Examiner(s): Black, Thomas G.
• Assistant Examiner(s): Choules, Jack M.
• Application Number:
  US08/336,331
• Time in Patent Office: 210 Days
• Field of Search: 364/DIG. 2, 395/325, 395/800
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A database server with a "shared nothing" system architecture has multiple nodes, each having its own central processing unit, primary and secondary memory for storing database tables and other data structures, and communication channels for communication with other ones of the nodes. The nodes are divided into first and second groups that share no resources. Each database table in the system is divided into fragments distributed for storage purposes over all the nodes in the system. To ensure continued data availability after a node failure, a "primary replica" and a "standby replica" of each fragment are each stored on nodes in different ones of the first and second groups. Database transactions are performed using the primary fragment replicas, and the standby replicas are updated using transaction log records. Every node of the system includes a data dictionary that stores information indicating where each primary and standby fragment replica is stored. The records of each database table are allocated as evenly as possible among the table fragments. A transaction manager on each node responds to database queries by determining which fragment of a database is being accessed by the query and then forwarding the database query to the node processor on which the primary replica of that fragment is stored. Upon failure of any one of the data processors in the system, each node updates the information in its data dictionary accordingly. In addition, the fragment replicas made unavailable by the node failure are regenerated and stored on the remaining available nodes in the same node group as the failed node.

Claims

1. A multiprocessor computer system, comprising:

N data processors, wherein N is a positive integer greater than three, each data processor having it own, separate, central processing unit, memory for storing database tables and other data structures, and communication channels for communication with other ones of said N data processors;

each of said N data processors independently executing a distinct instruction data stream;

at least a plurality of said N data processors including a communications processor for receiving transaction requests and for transmitting responses thereto;

said N data processors being divided into first and second groups, each having at least two data processors;

each data processor including;

fragmenting means for fragmenting each of said database tables into N fragments, and for storing a primary replica and a standby replica of each fragment, respectively, in different ones of said N data processors, wherein said different ones of said N data processors are in different ones of said first and second groups of data processors such that a complete copy of each of said database tables is located within each said group of data processors and such that simultaneous failure of all data processors in either of said groups wold leave a complete copy of each of said database tables in the other of said groups of data processors;

said fragmenting means adapted for allocating each record in any one of said database tables to a particular one of its N fragments in accordance with predefined criteria;

a data dictionary that stores information indicating where each said primary replica and standby replica of each fragment of said database tables is stored among said N data processors;

said fragmenting means further adapted for changing the information stored in said data dictionary upon failure of any one of said N data processors to indicate that the primary and standby replicas stored on the failed data processor are not available, for regenerating said primary and standby replicas on the failed data processor, and for storing portions of said regenerated replicas over non-failed ones, if any, of the data processors in the same group of data processors as the failed data processor; and

a transaction manager that responds to database queries by determining which fragment of a database table is being accessed by each database query and then forwarding said each database query to the data processor on which the primary replica of that database table fragment is stored.

2. The multiprocessor computer system of claim 1, wherein said first and second groups of data processors have different power supplies and different cooling systems.

3. The multiprocessor computer system of claim 1, wherein said allocation means allocates each record in any one of said database tables to a particular one of said one database table'"'"'s N fragments by hashing a primary key value of said record with a predefined hash function to obtain a fragment selection value.

4. The multiprocessor computer system of claim 1, wherein each said data processor includes a database manager for executing database queries, creating transaction log records while executing said database queries, said transaction log records indicating data table records changed by said database queries, forwarding a copy of each log record to the data processor on which is stored the standby replica of the database table fragment containing a corresponding changed data table record, and updating said standby replica of the database table fragment in accordance with said log record copy.

5. The multiprocessor computer system of claim 1,

wherein said fragmenting means in each data processor is adapted to respond to failure of one of said N data processors by (A) updating said data dictionary to indicate that the primary and standby fragment replicas on said one data processor are not available, (B) locating available ones of said primary and standby fragment replicas corresponding to the primary and standby fragment replicas on said one data processor, and (C) storing on a subset of said N data processors that have not failed, new replicas of the primary and standby fragment replicas made unavailable by said failure on said one data processor such that said primary and standby fragment replicas for each said database table fragments are stored in data processors in different ones of said groups of data processors.

6. The multiprocessor computer system of claim 5, wherein said fragmenting means is adapted to rebuild said primary and standby fragment replicas on said one data processor when said one data processor is restarted and to delete said new replicas stored on said subset of said N data processors.

7. The multiprocessor computer system of claim 5, wherein said fragment replicas are stored in units of pages, and said fragmenting means is adapted to store said new fragment replicas by sequentially reading the pages of said database table fragments that are being replicated, copying the read pages, transmitting the page copies to selected ones of said N data processors, and locking down said pages of said database table fragments only while they are being read so as not to block transactions utilizing said fragment replicas for which new replicase are being stored.

8. A multiprocessor computer system, comprising:

N data processors, wherein N is a positive integer greater than three, each data processor having its own, separate, central processing unit, memory for storing database tables and other data structures, and communication channels for communication with other ones of said N data processors;

each of said N data processors independently executing a distinct instruction data stream;

at least a plurality of said N data processors including a communications processor for receiving transaction requests and for transmitting responses thereto;

said N data processors being divided into first and second groups, each having at least two data processors;

each data processor including;

fragmenting means for fragmenting each of said database tables into N fragments, and for storing a primary replica and a standby replica of each fragment, respectively, in different ones of said N data processors, wherein said different ones of said N data processors are in different ones of said first and second groups of data processors such that a complete copy of each of said database tables is located within each said group of data processors and such that simultaneous failure of all data processors in either of said groups would leave a complete copy of each of said database tables in the other of said groups of data processors;

said fragmenting means adapted for allocating each record in any one of said database tables to a particular one of its N fragments in accordance with predefined criteria;

a data dictionary that store information indicating where each said primary replica and standby replica of each fragment of said database tables is stored among said N data processors;

said fragmenting means further adapted for changing the information stored in said data dictionary upon failure of any one of said N data processors to indicate that the primary and standby replicas stored on the failed data processor are not available, for generating new replicas of the primary and standby replicas made unavailable by the failure of said one data processor, for subdividing said new replicas into subfragments, and for distributing the storage of said subfragments in non-failed ones, if any, of the data processors in the same group of data processors as said one failed data processor; and

a transaction manager that responds to database queries by determining which fragment of a database table is being accessed by each database query and then forwarding said each database query to the data processor on which the primary replica of that database table fragment is stored.

9. A method of distributing data storage and transactional workloads in multiprocessor computer system having:

N data processors, wherein N is a positive integer greater than three, each data processor having its own, separate, central processing unit, memory for storing database tables and other data structures, and communication channels for communication with other ones of said N data processors;

at least a plurality of said N data processors including a communications processor that receives transaction requests and transmits responses thereto;

said N data processors being divided into first and second groups, each having at least two data processors;

the steps of the method comprising;

independently executing a distinct instruction data stream on each of said N data processors;

fragmenting each of said database tables into N fragments, and storing a primary replica and a standby replica of each fragment, respectively, in different ones of said N data processors, wherein said different ones of said N data processors are in different ones of said first and second groups of data processors such that a complete copy of each of said database tables is located within each said group of data processors and such that simultaneous failure of all data processors in either of said groups would leave a complete copy of each of said database tables in the other of said groups of data processors;

said fragmenting step including allocating each record in any one of said database tables to a particular one of its N fragments in accordance with predefined criteria;

storing in a data dictionary in each of said N data processors information indicating where each said primary replica and standby replica of each fragment of said database tables is stored among said N data processors;

upon failure of any one of said N data processors, changing the information stored in said data dictionary to indicate that the primary and standby replicas stored on the failed data processor are not available; and

responding to database queries by determining which fragment of a database table is being accessed by each database query, accessing the information in said data dictionary to determine which one of said N data processors contains the primary replica of that database table fragment, and then forwarding each said database query to said one of said N data processors.

10. The method of claim 9, wherein said allocating step allocates each record in any one of said database tables to a particular one of said one database table'"'"'s N fragments by hashing a primary key value of said record with a predefined hash function to obtain a fragment selection value.

11. The method of claim 9, wherein each said data processor executes database queries, creates transaction log records while executing said database queries, said transaction log records indicating data table records changed by said database queries, forwards a copy of each log record to the data processor on which is stored the standby replica of the database table fragment containing a corresponding changed data table record, and updates standby replicas of database table fragment corresponding to log record copies received by said each data processor.

12. The method of claim 9, further including responding to failure of one of said N data processors by (A) updating said data dictionary to indicate that the primary and standby fragment replicas on said one data processor are not available, (B) locating available ones of said primary and standby fragment replicas corresponding to the primary and standby fragment replicas on said one data processor, and (C) storing on a subset of said N data processors that have not failed, new replicas of the primary and standby fragment replicas made unavailable by said failure on said one data processor such that, after said new replicas are stored, said primary and standby fragment replicas for each said data database table fragment are stored in data processors in different ones of said groups of data processors.

13. The method of claim 12, further including rebuilding said primary and standby fragment replicas on said one data processor when said one data processor is restarted and deleting said new replicas stored on said subset of said N data processors.

14. The method of claim 12, including storing said fragment replicas in units of pages, said storing new replicas step including sequentially reading the pages of said database table fragments that are being replicated, copying the read pages, transmitting the page copies to selected ones of said N data processors, and locking down said pages of said database table fragments only while they are being read so as not to block transactions utilizing said fragment replicas for which new replicase are being stored.

15. A method of distributing data storage and transactional workloads in multiprocessor computer system having:

N data processors, wherein N is a positive integer greater than three, each data processor having its own, separate, central processing unit, memory for storing database tables and other data structures, and communication channels for communication with other ones of said N data processors;

at least a plurality of said N data processors including a communications processor that receives transaction requests and transmits responses thereto;

said N data processors being divided into first and second groups, each having at least two data processors;

the steps of the method comprising;

independently executing a distinct instruction data stream on each of said N data processors;

fragmenting each of said database tables into N fragments, and storing a primary replica and a standby replica of each fragment, respectively, in different ones of said N data processors, wherein said different ones of said N data processors are in different ones of said first and second groups of data processors such that a complete copy of each of said database tables is located within each said group of data processors and such that simultaneous failure of all data processors in either of said groups would leave a complete copy of each of said database tables in the other of said groups of data processors;

said fragmenting step including allocating each record in any one of said database tables to a particular one of its N fragments in accordance with predefined criteria;

storing in a data dictionary in each of said N data processors information indicating where each said primary replica and standby replica of each fragment of said database tables is stored among said N data processors;

upon failure of any one of said N data processors, changing the information stored in said data dictionary to indicate that the primary and standby replicas stored on the failed data processor are not available, generating new replicas of the primary and standby replicas made unavailable by failure of said one data processor, subdividing said new replicas into subfragments and distributing storage of said subfragments in non-failed ones, if any, of said data processors in the same group of data processors as said one failed data processor; and

responding to database queries by determining which fragment of a database table is being accessed by each database query, accessing the information in said data dictionary to determine which one of said N data processors contains the primary replica of that database table fragment, and then forwarding said each database query to said one of said N data processors.