Cluster of processing nodes with distributed global flash memory using commodity server technology

• US 9,047,351 B2
• Filed: 04/11/2011
• Issued: 06/02/2015
• Est. Priority Date: 04/12/2010
• Status: Expired due to Fees

• Current Assignee: SanDisk Technologies LLC (Western Digital Corporation)
• Original Assignee: SanDisk Enterprise IP LLC (Western Digital Corporation)
• Inventors: Riddle, Thomas A., Dinker, Darpan, Eckhardt, Andrew D., Koster, Michael J.
• Primary Examiner(s): Islam, Mohammad
• Assistant Examiner(s): LIU, MIKKA H
• Application Number:
  US13/084,502
• Publication Number:
  US 20120017037A1
• Time in Patent Office: 1,513 Days
• Field of Search: 711/103, 711/E12.008, 711/115, 711/130, 711141-146, 711147-148, 711/153
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24569 Query processing with adaptation to specific hardware, e.g. adapted for using GPUs or SSDs
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems

Abstract

Approaches for a distributed storage system that comprises a plurality of nodes. Each node, of the plurality of nodes, executes one or more application processes which are capable of accessing persistent shared memory. The persistent shared memory is implemented by solid state devices physically maintained on each of the plurality of nodes. Each the one or more application processes, maintained on a particular node, of the plurality of nodes, communicates with a shared data fabric (SDF) to access the persistent shared memory. The persistent shared memory comprises a scoreboard implemented in shared DRAM memory that is mapped to a persistent storage. The scoreboard provides a crash tolerant mechanism for enabling application processes to communicate with the shared data fabric (SDF).

Claims

1. A distributed storage system, comprising:

a plurality of nodes, wherein each node of the plurality of nodes executes one or more application processes that access persistent shared memory, wherein each of the one or more application processes is maintained on a respective node of the plurality of nodes;

the persistent shared memory, wherein the persistent shared memory is implemented by solid state devices maintained on the plurality of nodes;

a scoreboard implemented in the persistent shared memory, the scoreboard storing one or more scoreboard entries corresponding to a respective outstanding data access operation for accessing data stored in the persistent shared memory, each of the one or more scoreboard entries including;

data identifying a respective outstanding data access operation;

a thread ID identifying a thread, of the one or more application processes, that initiated the respective outstanding data access operation; and

a completion status associated with the respective outstanding data access operation,

a shared data fabric to enable the one or more application processes to access the persistent shared memory; and

a process in a first node of the plurality of nodes to update a first scoreboard entry of the one or more scoreboard entries, the first scoreboard entry corresponding to a first data access operation performed by an application process executed by the first node.

2. The distributed storage system of claim 1, wherein regions of the persistent shared memory are allocated by one process and deallocated by a different process.

3. The distributed storage system of claim 1, wherein reference counts are maintained for regions of the persistent shared memory, and wherein only unreferenced regions of the persistent shared memory may be deallocated.

4. The distributed storage system of claim 1, further comprising a communication mechanism to designate one or more regions of the persistent shared memory as non-pageable.

5. The distributed storage system of claim 1, including:

a process in the first node of the plurality of nodes to free the first scoreboard entry for use by a subsequent data access operation, wherein freeing the first scoreboard entry is in accordance with completion of the first data access operation.

6. A method for accessing data in a distributed storage system, the method comprising:

at a first node in a distributed storage system comprising a plurality of nodes, performing operations comprising;

executing one or more application processes that access persistent shared memory, wherein;

executing the one or more application processes includes performing a first data access operation,

the persistent shared memory is implemented by solid state devices maintained on the plurality of nodes, and

a shared data fabric enables the one or more application processes to access the persistent shared memory;

determining whether a data object is in a memory of the node;

in accordance with a determination that the data object is not in the memory of the node;

creating a first scoreboard entry to be stored in a scoreboard implemented in the persistent shared memory, wherein the scoreboard stores one or more scoreboard entries, including the first scoreboard entry, corresponding to a respective outstanding data access operation for accessing data stored in the persistent shared memory, and wherein the first scoreboard entry corresponds to the first data access operation, each of the one or more scoreboard entries including;

data identifying a respective outstanding data access operation;

a thread ID identifying a thread, of the one or more application processes, that initiated the respective outstanding data access operation; and

a completion status associated with the respective outstanding data access operation; and

updating the first scoreboard entry.

7. The method of claim 6, wherein regions of the persistent shared memory are allocated by one process and deallocated by a different process.

8. The method of claim 6, wherein reference counts are maintained for regions of the persistent shared memory, and wherein only unreferenced regions of the persistent shared memory may be deallocated.

9. The method of claim 6, wherein one or more regions of the persistent shared memory is designated as non-pageable.

10. The method of claim 6, further comprising:

creating a system message based on the first scoreboard entry;

sending, to a second node of the plurality of nodes, the system message; and

receiving, from the second node of the plurality of nodes, a response message containing the data object,

wherein updating the first scoreboard entry includes updating the respective completion status associated with the first data access operation.

11. The method of claim 6, further comprising:

in accordance with completion of the first data access operation, freeing the first scoreboard entry for use by a subsequent data access operation.