Sharing Data Fabric for Coherent-Distributed Caching of Multi-Node Shared-Distributed Flash Memory

• US 20090240869A1
• Filed: 08/25/2008
• Published: 09/24/2009
• Est. Priority Date: 03/20/2008
• Status: Active Grant

• Current Assignee: SanDisk Technologies LLC (Western Digital Corporation)
• Original Assignee: Schooner Information Technology, Inc. (Western Digital Corporation)
• Inventors: O'Krafka, Brian Walter, Koster, Michael John, Cohen, Earl T., McWilliams, Thomas M., Dinker, Darpan
• Granted Patent: US 8,732,386 B2
• Time in Patent Office: Days
• US Class Current: 711/103
• CPC Class Codes: G06F 12/0284 Multiple user address space allocation, e.g. using different base addresses interprocessor communication G06F15/163
• CPC Class Codes: G06F 12/0806 Multiuser, multiprocessor or multiprocessing cache systems
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/2022 Flash memory
• CPC Class Codes: G06F 2212/2542 Non-uniform memory access [NUMA] architecture
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A Sharing Data Fabric (SDF) causes flash memory attached to multiple compute nodes to appear to be a single large memory space that is global yet shared by many applications running on the many compute nodes. Flash objects stored in flash memory of a home node are copied to an object cache in DRAM at an action node by SDF threads executing on the nodes. The home node has a flash object map locating flash objects in the home node'"'"'s flash memory, and a global cache directory that locates copies of the object in other sharing nodes. Application programs use an applications-programming interface (API) into the SDF to transparently get and put objects without regard to the object'"'"'s location on any of the many compute nodes. SDF threads and tables control coherency of objects in flash and DRAM.

Claims

1. A global shared flash-memory system comprising:

a plurality of compute nodes, each compute node in the plurality of compute nodes comprising;

a flash memory for storing homed objects in non-volatile solid-state memory cells;

a processor for executing an application program and threads;

a volatile memory, coupled to the processor, for storing cached objects, wherein the cached objects are copies of remote homed objects stored at remote compute nodes in the plurality of compute nodes;

a network interface controller (NIC), coupled by a network to other compute nodes, and responsive to the processor, that sends local copies of the homed objects stored in the flash memory to a first compute node in the plurality of compute nodes, and that receives from a second compute node in the plurality of compute nodes remote copies of the remote homed objects stored in the flash memory of the second compute node;

a Sharing Data Fabric (SDF) component executing on the processor, that enables storing the remote copies of the remote homed objects received by the NIC into the volatile memory as the cached objects;

wherein the application program executing on the processor accesses the cached objects from the volatile memory, including cached objects that are remote copies of the remote homed objects stored in the flash memory of the remote compute nodes in the plurality of compute nodes;

wherein a plurality of the SDF component, executing on a plurality of the processor in the plurality of compute nodes, copy the remote homed objects from a plurality of the flash memory in the plurality of compute nodes through the network for storage in the volatile memory as the cached objects,

wherein the remote homed objects from the flash memory are copied through the network for storage in the volatile memory as cached objects.

2. The global shared flash-memory system of claim 1 wherein each compute node in the plurality of compute nodes further comprises:

a local switch, coupled to the processor, the flash memory, and the NIC, for allowing the processor to communicate with other compute nodes through the NIC, and for accessing the homed objects in the flash memory to generate the local copies of the homed objects for transfer to the NIC for sending to the first compute node.

3. The global shared flash-memory system of claim 2 wherein the local switch is a Peripheral Components Interconnect Express (PCIe) switch that connects to the flash memory and to the NIC using a PCIe bus;

and

wherein the network is an Ethernet network that connects together the plurality of compute nodes.

4. The global shared flash-memory system of claim 1 further comprising:

an applications-programming interface (API) to the SDF component, the API being used by the application program to get a requested cached object that is not present in the volatile memory, the SDF component getting a remote copy of a homed object stored in the flash memory of a remote home node in the plurality of compute nodes and storing the remote copy as the requested cached object for access by the application program;

wherein the application program is not aware that the requested cached object is located at the remote home node;

whereby the application program uses the API to activate the SDF component to transparently retrieve objects from the remote home node.

5. The global shared flash-memory system of claim 1 wherein each compute node in the plurality of compute nodes further comprises:

a plurality of flash memory chips, wherein the flash memory comprises the plurality of flash memory chips;

a plurality of flash controllers that activate the plurality of flash memory chips to read and write flash blocks in the plurality of flash memory chips;

a flash manager for assigning the flash blocks for writing using a wear-leveling scheme to reduce wear of the flash blocks storing the homed objects,

whereby the flash blocks storing the homed objects are wear-leveled to reduce wear.

6. The global shared flash-memory system of claim 1 wherein each of the plurality of compute nodes acting as a home node further comprises:

a global cache directory having a sharing list, the sharing list being a list of ones of the plurality of compute nodes storing the cached objects that are copies of the homed objects stored in the flash memory of the home node;

a flash object map being used by the SDF component to locate the homed objects within the flash memory in response to a request for the cached objects from another compute node in the plurality of compute nodes,

whereby the homed objects are located with the flash object map and compute nodes storing the remote copies of the homed objects are located using the global cache directory.

7. The global shared flash-memory system of claim 6 further comprising:

coherency means, in the SDF component, for fetching a modified copy of a remote homed object from a sharing node in the plurality of compute nodes identified by the global cache directory, and for sending the modified copy instead of the remote homed object in response to a request for the cached object,

whereby coherency is maintained by fetching the modified copy from the sharing node.

8. The global shared flash-memory system of claim 1 wherein each of the plurality of compute nodes acting as an action node with a requested cached object further comprises:

a transaction table that stores a list of objects touched by a transaction and an initial state for all objects touched by the transaction;

wherein the initial state is a state before the transaction begins;

transaction recovery means for aborting the transaction by restoring the objects touched by the transaction to their initial state stored in the transaction table;

wherein the transaction is a series of operations that are all committed together or all aborted,

whereby transactions are aborted using the transaction table for recovery.

9. The global shared flash-memory system of claim 8 wherein each of the plurality of compute nodes acting as the action node with the requested cached object further comprises:

transaction-commit means for updating the homed objects, activated after all operations in the transaction are successfully completed and all prior outstanding requests for objects touched by the transaction are completed;

flash-update means, activated by the transaction-commit means, for sending modified objects touched by the transaction to home nodes in the plurality of compute nodes, the home nodes updating the homed objects in the flash memory using the modified objects with changes from the transaction,

whereby transactions are committed by updating the homed objects at home nodes once the action node indicates that all prior outstanding requests have successfully completed for all objects touched by the transaction.

10. The global shared flash-memory system of claim 1 wherein the non-volatile solid-state memory cells consume less power than a rotating disk when writing the homed objects,

whereby power consumption per sector of data written is reduced using the global shared flash-memory system than a rotating-disk-storage system.

11. The global shared flash-memory system of claim 10 wherein the flash memory stores objects as non-volatile data stored in electrically-erasable programmable read-only memory (EEPROM) memory cells or in phase-change memory (PCM) cells.

12. A Sharing Data Fabric (SDF) in a multi-processor distributed flash-memory system comprising:

a plurality of SDF threads executing on a plurality of processors on a plurality of storage nodes;

a global shared flash memory that comprises flash memory storage units at each of the plurality of storage nodes;

a volatile memory at each of the storage nodes and accessible by a local processor at the storage node;

an interconnect that connects the plurality of storage nodes in a network;

a messaging interface that sends and receives messages over the interconnect;

an object cache in the volatile memory at an action node having a local processor executing an application program that accesses objects in the object cache;

a request messager in the action node, activated when the application program requests a requested object that is not present in the object cache, the request messager generating a home-node name of a home node in the plurality of storage nodes and sending a request message with a requested-object identifier of the requested object to the home node over the interconnect using the messaging interface;

a responding SDF thread in the plurality of SDF threads, the responding SDF thread running on a remote processor on the home node, the responding SDF thread responding to the request message sent by the request messager at the action node;

a flash object map stored at the home node, the flash object map receiving the requested-object identifier extracted from the request message by the responding SDF thread and generating a location identifier;

the responding SDF thread using the location identifier generated by the flash object map to access a homed object stored in a flash memory storage unit of the home node, the responding SDF thread sending a copy of the homed object to the action node in a response message sent over the interconnect using the messaging interface; and

a cache loader at the action node, receiving the response message from the home node, the cache loader writing the copy of the homed object to the object cache in the volatile memory of the action node, the application program accessing the copy of the homed object in the object cache,

whereby objects stored in the flash memory storage unit of the home node are sent to the action node for caching in the volatile memory.

13. The SDF of claim 12 further comprising:

a global cache directory in the volatile memory at the home node, the global cache directory storing a list of sharing nodes in the plurality of storage nodes, the sharing nodes storing a cached copy of the homed object;

wherein the responding SDF thread does not read the flash memory storage unit of the home node and instead sends a forwarding message to the sharing node when the global cache directory indicates that the sharing node stores the cached copy; and

a forwarding SDF thread in the plurality of SDF threads, the forwarding SDF thread running on a remote processor on the sharing node, the forwarding SDF thread responding to the forwarding message sent by the responding SDF thread at the home node, the forwarding SDF thread reading the cached copy from the volatile memory at the sharing node and sending the cached copy to the action node or to the home node,

whereby coherency is maintained using the global cache directory to locate the sharing node with the cached copy.

14. The SDF of claim 12 wherein the cache loader at the action node is a cache-loader SDF thread in the plurality of SDF threads,

wherein the application program is not halted while the homed object is read from the flash memory storage unit of the home node;

whereby the copy of the homed object is loaded into the object cache asynchronously.

15. The SDF of claim 12 further comprising:

an applications-programming interface (API), activated by the application program at the action node, for commanding the request messager in the action node to GET the homed object from the home node for read access, to PUT the homed object back into the flash memory storage unit of the home node from the object cache of the action node when write access is completed, to lock the requested object to prevent access by other storage nodes, and to unlock the requested object to allow access by other storage nodes.

16. A computer-implemented method for transparently sharing objects in a distributed shared flash memory comprising:

generating a call to a Sharing Data Fabric (SDF) from an application program when an object requested by the application program is not present in an object cache for a processor on an action node executing the application program;

the SDF generating a home-node name of a home node from an object identifier of the object;

sending a request-object message from the action node to the home node identified by the home-node name;

searching a global cache directory at the home node with the object identifier of the object to find a sharing entry when the object is cached at a sharing node;

when the global cache directory indicates that the object is not cached at the sharing node;

reading a flash object map to obtain a flash location of a flash object having a same object identifier as the object in a flash memory controlled by the home node;

reading the flash object from the flash memory of the home node and sending a message to the action node with the flash object;

wherein the flash memory is a non-volatile solid-state memory and wherein the object cache is in a volatile solid-state memory;

when the global cache directory indicates that the object is cached at the sharing node;

reading the sharing entry to obtain a sharing-node name of the sharing node;

sending a sharing message from the home node to the sharing node;

reading an object copy from an object cache at the sharing node in response to the sharing message being received at the sharing node;

wherein the object copy has the object identifier;

sending a message to the action node with the object copy from the sharing node directly or through the home node;

loading the object copy or the flash object into the object cache at the action node, as the object for use by the application program; and

updating the sharing entry in the global cache directory at the home node to indicate that the action node stores the object copy in its object cache,

wherein the flash object from flash memory at the home node is copied to the object cache at the action node by the SDF without the application program using the home-node name of the home node having the flash object;

wherein the home node is selected from a plurality of nodes each storing flash objects, each node having a flash memory controlled by a local processor of the node,

whereby objects are transparently fetched from flash memory at the home node.

17. The computer-implemented method of claim 16 wherein generating the call to the SDF comprises generating an applications-programming interface (API) call from the application program to the SDF.

18. The computer-implemented method of claim 16 further comprising:

when the application program generates the call in a transaction of operations that must all be executed together or aborted together;

writing the object identifier to a transaction table at the action node for each object touched in the transaction;

writing an initial state of the object to the transaction table at the action node for each object touched in the transaction;

when the transaction is aborted, using the initial state of the object stored in the transaction table to return the object to the initial state for each object touched in the transaction;

whereby objects are returned to initial states stored in the transaction table for aborted transactions.

19. The computer-implemented method of claim 18 further comprising:

when all operations in a transaction are successfully completed;

reading a transient protocol state table at the action node to identify objects touched by the transaction;

waiting for outstanding requests to complete for objects touched by the transaction;

when all outstanding requests have completed, sending modified objects touched by the transaction to home nodes to update flash objects with changes from the transaction,

whereby transactions are committed by updating flash objects at home nodes once the action node indicates that all outstanding requests have successfully completed for all objects touched by the transaction.

20. The computer-implemented method of claim 16 further comprising:

the SDF sending a lock-request message from the action node to the home node when the application program requests a lock for an object from the SDF;

searching a global cache directory at the home node with the object identifier of the object to find the sharing entry when the object is cached at the sharing node;

waiting until the object is free in the global cache directory and no longer locked by the sharing node and then sending a return-lock message to the action node and indicating in the global cache directory that the sharing entry is locked.