Mechanism for persisting messages in a storage system

• US 9,003,144 B1
• Filed: 06/04/2014
• Issued: 04/07/2015
• Est. Priority Date: 06/04/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Hayes, John, Lee, Robert, Gupta, Shantanu, Davis, John, Gold, Brian, Ostrovsky, Igor, Sears, Rusty
• Primary Examiner(s): CARDWELL, ERIC
• Application Number:
  US14/296,146
• Time in Patent Office: 307 Days
• US Class Current: 711/162
• CPC Class Codes: G06F 11/1016 Error in accessing a memory location, i.e. addressing error
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 12/0638 Combination of memories, e.g. ROM and RAM such as to permit replacement or supplementing of words in one module by words in another module address formation of the next microinstruction G06F9/26; masking faults in memories by using spares or by reconfiguring G11C29/70
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 2212/205 Hybrid memory, e.g. using both volatile and non-volatile memory
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

A plurality of storage nodes in a single chassis is provided. The plurality of storage nodes in the single chassis is configured to communicate together as a storage cluster. Each of the plurality of storage nodes includes nonvolatile solid-state memory for user data storage. The plurality of storage nodes is configured to distribute the user data and metadata associated with the user data throughout the plurality of storage nodes such that the plurality of storage nodes maintain the ability to read the user data, using erasure coding, despite a loss of two of the plurality of storage nodes. The plurality of storage nodes configured to initiate an action based on the redundant copies of the metadata, responsive to achieving a level of redundancy for the redundant copies of the metadata. A method for accessing user data in a plurality of storage nodes having nonvolatile solid-state memory is also provided.

Claims

1. A plurality of storage nodes in a single chassis, comprising:

the plurality of storage nodes communicating together as a storage cluster;

each of the plurality of storage nodes having at least one authority, and nonvolatile solid-state memory for storage of user data and metadata, wherein each authority controls a portion of the user data and determines which type of erasure coding scheme is applied to the portion of the user data independent of further authorities;

the plurality of storage nodes configured to distribute the user data and redundant copies of metadata regarding a message throughout the plurality of storage nodes such that the plurality of storage nodes can read the user data, using erasure coding, despite loss of two of the plurality of storage nodes; and

the plurality of storage nodes configured to initiate an action based on the redundant copies of the metadata, such that a source authority of one of the plurality of storage nodes receives the message, records the message redundantly throughout the plurality of storage nodes, then delivers the message to a destination authority of a further one of the storage nodes responsive to achieving a level of redundancy for the redundant copies of the metadata regarding the message, wherein the destination authority acts upon the message.

2. The plurality of storage nodes of claim 1, wherein the action includes one of a data recovery, or a data rebuild, wherein the single chassis has a communications interconnect that couples the plurality of storage nodes, and wherein at least a portion of the communications interconnect is integral with the single chassis.

3. The plurality of storage nodes of claim 1, further comprising:

each of the plurality of storage nodes having a capacitor coupled to a RAM (random access memory), the capacitor having sufficient capacity to power the RAM for a predefined time period in event of a power loss.

4. The plurality of storage nodes of claim 1, wherein the single chassis is an enclosure with internal power distribution and an internal communication bus coupling the plurality of storage nodes, and wherein the single chassis couples the plurality of storage nodes to an external communication bus.

5. The plurality of storage nodes of claim 1, further comprising:

each of the plurality of storage nodes having a direct memory access (DMA) unit configured to copy the metadata from a RAM (random access memory) to the solid-state memory, responsive to a power loss.

6. The plurality of storage nodes of claim 1, further comprising:

each of the plurality of storage nodes configured to send a copy of the metadata to others of the plurality of storage nodes.

7. The plurality of storage nodes of claim 1, further comprising:

each of the plurality of storage nodes having decision logic configured to initiate the action in cooperation with others of the plurality of storage nodes.

8. A storage cluster, comprising:

a plurality of storage nodes within a single chassis;

each of the plurality of storage nodes having nonvolatile solid-state memory for storage of user data;

each of the plurality of storage nodes having at least one authority, each authority configured to determine which type of erasure coding scheme is applied to a portion of user data owned by the authority, independent of further authorities of the plurality of storage nodes;

the plurality of storage nodes configured to distribute the user data and redundant copies of metadata regarding a message throughout the plurality of storage nodes via an internal communications interconnect of the single chassis, the redundant copies of the metadata regarding the message stored in at least a subset of the plurality of storage nodes, the plurality of storage nodes are able to access the user data, via erasure coding, with a failure of two of the plurality of storage nodes; and

the plurality of storage nodes configured to determine what action to carry to completion, based on the redundant copies of the metadata, and further based on a source authority of one of the plurality of storage nodes receiving the message, recording the message metadata regarding the message redundantly throughout the plurality of storage nodes and then delivering the message to a destination authority of a further one of the plurality of storage nodes responsive to achieving a level of redundancy for the redundant copies of the metadata regarding the message, whereupon the destination authority acts upon the message.

9. The storage cluster of claim 8, wherein the solid-state memory includes flash, and wherein the action includes one of reading the user data, writing the user data, updating a property of the user data, recovering the user data, or rebuilding the user data.

10. The storage cluster of claim 8, further comprising:

each of the plurality of storage nodes having decision logic for determining what action to carry to completion, the decision logic including one of witnessing logic, voting logic, or consensus logic.

11. The storage cluster of claim 8, further comprising:

each of the plurality of storage nodes having a nonvolatile random access memory (NVRAM), configured to store the metadata.

12. The storage cluster of claim 8, further comprising:

each of the plurality of storage nodes having random-access memory (RAM) and a direct memory access (DMA) unit configured to move the metadata from the RAM to the solid-state memory, responsive to a power failure.

13. The storage cluster of claim 8, further comprising:

each of the plurality of storage nodes configured such that the redundant copies of metadata are communicated via one of an interconnect switch configured to broadcast a message to each of the plurality of storage nodes, one of the plurality of storage nodes sending a copy of the metadata to others of the plurality of storage nodes, or one of the plurality of storage nodes sending a copy of a message to others of the plurality of storage nodes.

14. A method for persisting messages in a plurality of storage nodes having nonvolatile solid-state memory, comprising:

distributing user data throughout the plurality of storage nodes through erasure coding, wherein the plurality of storage nodes are housed within a single chassis that couples the plurality of storage nodes as a storage cluster, and wherein the user data is accessible, via the erasure coding, in the plurality of storage nodes in event of two of the plurality of storage nodes being unreachable;

receiving a message at a source authority of one of the plurality of storage nodes;

storing metadata relating to the message in the one of the plurality of storage nodes;

distributing redundant copies of the metadata relating to the message from the source authority to others of the plurality of storage nodes;

determining whether a level of redundancy for distributed redundant copies of the metadata regarding the message has been achieved; and

determining an action, including the source authority delivering the message to a destination authority of a further one of the plurality of storage nodes and having the destination authority act upon the message, based on the redundant copies of the metadata responsive to achieving the level of redundancy, wherein a processor performs at least one method operation.

15. The method of claim 14, further comprising:

broadcasting the metadata to the one of the plurality of storage nodes and to the others of the plurality of storage nodes.

16. The method of claim 14, wherein each of the plurality of storage nodes has flash memory as the solid-state memory, and wherein the distributing user data is via a communications interconnect that is integral with the single chassis.

17. The method of claim 14, further comprising:

detecting a power loss; and

supplying electrical power from an energy reserve sufficient to sustain the metadata during a direct memory access (DMA) to copy the metadata.

18. The method of claim 14, further comprising:

copying the metadata from random access memory to a flash memory of the one of the plurality of storage nodes, in response to a power loss, wherein the flash memory is included in the solid-state memory.

19. The method of claim 14, further comprising:

delivering the message to one of the others of the plurality of storage nodes after determining the level of redundancy has been achieved.

20. The method of claim 14, wherein the metadata includes one of a request to lock a file, a request to unlock a file, a change in a permission of a file, an update to a directory structure, a request to write a file that has executable instructions, a request to write a file name that is reserved, an update to a fingerprint data, an update to a hash table, or an update to a log, and wherein the action is based on the metadata.