Scalable Database Management Software on a Cluster of Nodes Using a Shared-Distributed Flash Memory

• US 20090240664A1
• Filed: 11/24/2008
• Published: 09/24/2009
• Est. Priority Date: 03/20/2008
• Status: Active Grant

• Current Assignee: SanDisk Technologies LLC (Western Digital Corporation)
• Original Assignee: Schooner Information Technology, Inc. (Western Digital Corporation)
• Inventors: Ouye, Darryl Manabu, Eckhardt, Andrew David, Cohen, Earl T., McWilliams, Thomas M., Dinker, Darpan, O'Krafka, Brian Walter
• Granted Patent: US 8,229,945 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0842 for multiprocessing or multitasking
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2212/2022 Flash memory
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A distributed database system has multiple compute nodes each running an instance of a database management system (DBMS) program that accesses database records in a local buffer cache. Records are persistently stored in distributed flash memory on multiple storage nodes. A Sharing Data Fabric (SDF) is a middleware layer between the DBMS programs and the storage nodes and has API functions called by the DBMS programs when a requested record is not present in the local buffer cache. The SDF fetches the requested record from flash memory and loads a copy into the local buffer cache. The SDF has threads on a home storage node that locate database records using a node map. A global cache directory locks and pins records to local buffer caches for updating by a node'"'"'s DBMS program. DBMS operations are grouped into transactions that are committed or aborted together as a unit.

Claims

1. A distributed multi-node database comprising:

a plurality of compute nodes, each compute node comprising;

a compute processor for executing threads and programs;

a database management system (DBMS) program executing on the compute processor, the DBMS program receiving a database query from a user, the database query specifying a target database record;

a Sharing Data Fabric (SDF) thread executing on the compute processor;

a local buffer cache accessed by the DBMS program, the local buffer cache storing a cached database record of the target database record that is accessed by the DBMS program;

a plurality of storage nodes, each storage node comprising;

a non-volatile memory for storing a portion of a database comprising persistent database records;

a storage processor executing a storage thread;

wherein the non-volatile memory is a solid-state memory that retains data when power is disconnected; and

a physical interconnect that connects the plurality of compute nodes to the plurality of storage nodes;

wherein the SDF thread is activated by the DBMS program when the target database record is not present in the local buffer cache;

wherein the SDF thread further is enabled to identify a target storage node in the plurality of storage nodes, the target storage node storing the target database record in the portion of the database stored on the target storage node;

wherein the storage thread further is enabled to read the target database record from the non-volatile memory of the target storage node and send the target database record over the physical interconnect to the SDF thread on the compute node for storage in the local buffer cache as the cached database record,

whereby the DBMS program on the compute node processes the database query using the SDF thread to fetch the target database record from the target storage node using the storage thread.

2. The distributed multi-node database of claim 1 wherein the DBMS program executing on the compute processor accesses the database stored on the plurality of storage nodes using the SDF thread executing on the compute processor, the SDF thread communicating with the storage thread executing on the storage processor to load the cached database record into the local buffer cache;

wherein the DBMS program is enabled to execute in an address space on a single node;

wherein the SDF thread makes the database visible within the address space to the DBMS program;

wherein the database appears to the DBMS program to be stored on the compute node of the DBMS program, wherein the SDF thread communicating with the storage thread transparently access the database stored on the plurality of storage nodes,

whereby the DBMS program transparently accesses the database stored on the plurality of storage nodes using the SDF thread.

3. The distributed multi-node database of claim 2 wherein a plurality of the DBMS program executing on the compute processor on the plurality of compute nodes simultaneously access the database stored on the plurality of storage nodes in the address space,

whereby multiple DBMS programs access the database in the address space.

4. The distributed multi-node database of claim 1 wherein each compute node is also a storage node, wherein the compute processor and the storage processor are a shared processor that executes the DBMS program and the SDF thread;

wherein the target database record is read for the SDF thread from the non-volatile memory when the target storage node is the compute node of the DBMS program processing the database query,

whereby the SDF thread directly reads the non-volatile memory when the target storage node is a local node of the DBMS program.

5. The distributed multi-node database of claim 1 wherein the DBMS program further comprises:

a query process, executing on the compute processor, for processing the database query to identify the target database record; and

a storage manager, executing on the compute processor, for identifying the cached database record from the local buffer cache when the cached database record is a copy of the target database record;

wherein the storage manager activates the SDF thread to retrieve the target database record and load a copy of the target database record into the local buffer cache when the target database record misses in the local buffer cache.

6. The distributed multi-node database of claim 5 wherein the DBMS program further comprises:

a log writer that sends a log message to a log storage node in the plurality of storage nodes;

wherein a logical sequence number is associated with the log message stored in the log file,

whereby operations performed by the DBMS program are logged at the log storage node.

7. The distributed multi-node database of claim 5 wherein the database stored on the plurality of storage nodes comprises:

database tables that comprise a plurality of database records including the target database record;

database indices that index the database tables; and

log files that log operations performed on the database.

8. The distributed multi-node database of claim 1 wherein the compute node further comprises:

a transaction table that stores a list of database records modified by a transaction and an initial state for database records touched by the transaction;

transaction recovery means for aborting the transaction by restoring the database records modified by the transaction to their initial state stored in the transaction table;

wherein the transaction is a series of operations performed by the DBMS program that are all committed together or all aborted together,

whereby transactions are aborted using the transaction table.

9. The distributed multi-node database of claim 8 wherein the compute node further comprises:

transaction-commit means for updating a database record, activated when all operations in the transaction are successfully completed and all outstanding requests for database records modified by the transaction are completed;

persistent-update means, activated by the transaction-commit means, for sending the database records modified by the transaction to home nodes in the plurality of storage nodes, the home nodes updating persistent database records in the non-volatile memory using the modified database records with changes from the transaction,

whereby transactions are committed by updating persistent database records at home nodes once all outstanding requests have successfully completed for all database records modified by the transaction.

10. The distributed multi-node database of claim 1 further comprising:

an applications-programming interface (API) to the SDF thread, the API being called by the DBMS program to get the target database record that is not present in the local buffer cache, the SDF thread communicating with the storage thread on the target storage node to copy the target database record into the local buffer cache as the cached database record read by the DBMS program,

wherein the DBMS program is not aware that the target database record is located at the target storage node;

whereby the DBMS program uses the API to activate the SDF thread to transparently copy the target database record from the target storage node.

11. The distributed multi-node database of claim 10 wherein the API to the SDF thread comprises a multiprocessing interface, a mutual-exclusion MUTEX library routine, or a shared container of synchronization primitives including test-and-set or compare-and-swap.

12. The distributed multi-node database of claim 1 wherein the storage node further comprises:

a global cache directory having a sharing list, the sharing list being a list of sharing nodes in the plurality of compute nodes that store the cached database record that is a copy of the target database record stored in the non-volatile memory of the storage node;

a database record map that is used by the storage thread to locate the target database record within the non-volatile memory in response to a request for the target database record from the compute node,

whereby the target database record is located with the database record map and compute nodes storing copies of the target database record are located using the global cache directory.

13. The distributed multi-node database of claim 1 wherein the non-volatile memory stores database records in electrically-erasable programmable read-only memory (EEPROM) memory cells or in phase-change memory (PCM) cells.

14. The distributed multi-node database of claim 1 wherein the non-volatile memory consumes less power than a rotating disk when accessing the database,

whereby power consumption is reduced.

15. A computer-program product comprising:

a computer-readable storage medium having computer-executable program code means tangibly embodied therein, which when executed on a computing device manages a distributed database stored across multiple nodes of flash memory, the computer-executable storage program code means in the computer-program product comprising;

query processing means for processing a database query received from a user, the database query specifying a target database record in a database;

local buffer cache means for storing cached database records in a volatile memory;

table reader means for reading the target database record from the local buffer cache means when a hit occurs;

table writer means for writing to the target database record in the local buffer cache means when a hit occurs and the target database record is in the local buffer cache means;

Sharing Data Fabric (SDF) thread means, activated by a GET command from the table reader means when the hit does not occur in the local buffer cache means, for sending a request message over an interconnect from a first node to a target storage node in a plurality of nodes, and for identifying the target storage node using an identifier of the target database record;

storage thread means, executing on the target storage node, for reading the target database record from the flash memory at the target storage node that stores a portion of the database, and for sending the target database record over the interconnect to the SDF thread means on the first node; and

miss loader means, in the SDF thread means, for receiving the target database record from the target storage node over the interconnect, and for writing the target database record into the local buffer cache means, and for activating the table reader means to read the target database record as a hit,

whereby the target database record is retrieved from the target storage node using the SDF thread means transparently to the table reader means.

16. The computer-program product of claim 15 wherein the computer-executable program code means further comprises:

flush means, in the SDF means, for sending a flush message to the target storage node, the flush message communicating the target database record modified by the table writer means;

persistent store means, in the storage thread means, for writing the target database record to the flash memory at the target storage node in response to the flush message,

whereby the target database record modified by the table writer means is flushed to flash memory at the target storage node.

17. The computer-program product of claim 15 wherein the computer-executable program code means further comprises:

lock request means, in the SDF thread means, for sending a lock message to the target storage node in response to the table writer means attempting to write to the target database record in the local buffer cache means;

directory means, on the target storage node, for indicating that the target database record is locked by the first node in response to the lock message, the directory means preventing other nodes from writing to the target database record when locked;

unlock means, in the SDF thread means, for sending an unlock message to the target storage node in response to the table writer means finishing writing to the target database record in the local buffer cache means;

wherein the directory means, on the target storage node, indicates that the target database record is not locked by the first node in response to the unlock message,

whereby database records are locked for writing.

18. The computer-program product of claim 17 wherein the computer-executable program code means further comprises:

sync means, in the SDF means, for sending a sync message to the target storage node;

wherein the storage thread means further comprises sync update means for allowing other nodes in the plurality of nodes to access the target database record modified by the table writer means by updating the directory means on the target storage node;

cache pin means, in the SDF means, for marking for retention the target database record in the local buffer cache means during a current transaction.

19. A transactional distributed database comprising:

a plurality of database management system (DBMS) programs executing on a plurality of processors on a plurality of compute nodes, each DBMS program receiving database queries from clients that identify target data items in a database;

a plurality of storage nodes each having a non-volatile memory that stores a portion of the database, wherein the database is partitioned across the plurality of storage nodes;

wherein the non-volatile memory is a solid-state memory that retains data when power is disconnected;

an interconnect that passes messages between the plurality of compute nodes and the plurality of storage nodes; and

a plurality of Sharing Data Fabric (SDF) threads executing on the plurality of processors on the plurality of compute nodes, a SDF thread being activated by the DBMS program when the target data items are not present in a local buffer cache, the SDF thread identifying a target storage node in the plurality of storage nodes, the target storage node being a home node that persistently stores the target data items, the SDF thread sending a request message to the target storage node;

wherein each storage node further comprises a processor executing a storage thread, the storage thread reading the target data items from the non-volatile memory in response to the request message received over the interconnect, the storage thread sending the target data items over the interconnect in a reply message sent in response to the request message;

wherein each of the plurality of DBMS programs operates in a shared address space that includes the database stored on the plurality of storage nodes, the plurality of SDF threads transparently accessing data items stored on the plurality of storage nodes.

20. The transactional distributed database of claim 19 wherein an action node in the plurality of nodes comprises:

a transaction table, wherein operations performed by the DBMS program are grouped into a transaction having a transaction entry in the transaction table,

wherein a first operation in a new transaction causes the SDF thread to create a new entry in the transaction table for the new transaction;

wherein the SDF thread adds a list of touched data items to the new entry in the transaction table when subsequent operations in the transaction access the touched data items in the database;

wherein a final commit operation in the new transaction causes the SDF thread to flush modified data items in the list of touched data items in the new entry, wherein modified data items are sent over the interconnect to the plurality of storage nodes for storage in the non-volatile memory of storage nodes storing the modified data items;

wherein when an error occurs during processing of the new transaction, an abort operation is processed instead of the final commit operation, the abort operation preventing any of the modified data items from updating the non-volatile memory in the plurality of storage nodes,

whereby operations in the new transaction are committed together or aborted together as an atomic transaction.