Storage system architecture

• US 9,967,342 B2
• Filed: 12/12/2016
• Issued: 05/08/2018
• Est. Priority Date: 06/04/2014
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: Pure Storage, Inc.
• Inventors: Colgrove, John, Davis, John D., Hayes, John
• Primary Examiner(s): ELMORE, STEPHEN C
• Application Number:
  US15/376,220
• Publication Number:
  US 20170093980A1
• Time in Patent Office: 512 Days
• Field of Search: 711103, 711154, 711156, 711158, 711170, 713340, 714 62, 714 624, 714 632
• CPC Class Codes: G06F 11/1068 in sector programmable memories, e.g. flash disk G06F11/1072 takes precedence
• CPC Class Codes: G06F 11/108 Parity data distribution in semiconductor storages, e.g. in SSD
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/845 Systems in which the redundancy can be transformed in increased performance
• CPC Class Codes: G06F 2212/7206 Reconfiguration of flash memory system
• CPC Class Codes: G06F 2212/7207 management of metadata or control data
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 : Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: G06F 3/0688 : Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 : Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: G11C 29/52 : Protection of memory contents; Detection of errors in memory contents
• CPC Class Codes: H03M 13/154 : Error and erasure correction, e.g. by using the error and erasure locator or Forney polynomial
• CPC Class Codes: H04L 12/40169 : Flexible bus arrangements arrangements for maintenance or administration involving management of faults; events, alarms H04L41/06; automatic restoration of network faults H04L41/0654
• CPC Class Codes: H04L 49/10 : characterised by the switching fabric construction
• CPC Class Codes: H04L 49/111 : Switch interfaces, e.g. port details
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/51 : Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

A storage system is provided. The storage system includes a plurality of storage units, each of the plurality of storage units having storage memory for user data and a plurality of storage nodes, each of the plurality of storage nodes configured to have ownership of a portion of the user data. The storage system includes a first pathway, coupling the plurality of storage units such that each of the plurality of storage units can communicate with at least one other of the plurality of storage units via the first pathway without assistance from the plurality of storage nodes.

Claims

1. A storage system comprising:

a plurality of storage nodes, each having a processor;

a plurality of storage units, each having a controller and solid-state storage memory;

a first pathway that couples the processors of the plurality of storage nodes and is configured to couple to a network external to the storage system; and

a second pathway that couples at least a first controller of a first storage unit to a second controller of a second storage unit for communication, wherein the second pathway is a direct controller to controller coupling.

2. The storage system of claim 1, further comprising:

the first controller of the first storage unit configured to send a write request and a data shard or a parity shard through the second pathway to the second controller of the second storage units; and

the second controller of the second storage unit configured to write the data shard or the parity shard into the solid-state storage memory of the second storage unit, responsive to the write request.

3. The storage system of claim 1, further comprising:

the first controller of the first storage unit configured to send an inquiry through the second pathway to the second controller of the second storage unit; and

the second controller of the second storage unit configured to send a reply, through the second pathway to the first controller of the first storage unit, responsive to the inquiry, with the processors of the plurality of storage nodes being non-participatory in the communication.

4. The storage system of claim 1, further comprising:

the first controller of the first storage unit configured to send a read request for data or metadata through the second pathway to the second controller of the second storage unit; and

the second controller of the second storage unit configured to send the data or the metadata through the second pathway to the first controller of the second storage unit, responsive to the read request.

5. The storage system of claim 1, further comprising:

the first or second pathway coupling at least a first processor of a first storage node to at least the first controller of the first storage unit;

the first processor of the first storage node configured to send a direction to construct data from data shards, through the first or second pathway to the first controller of the first storage unit; and

the first controller of the first storage unit configured to send requests for the data shards through the second pathway to further controllers of further storage units.

6. The storage system of claim 1, further comprising:

each of the plurality of storage units configured to communicate a heartbeat through the second pathway.

7. The storage system of claim 1, further comprising:

the first pathway including a network switch; and

the second pathway including a PCI (peripheral component interconnect) Express bus.

8. A storage system comprising:

a plurality of storage nodes having a plurality of processors;

a plurality of storage units, each one of the plurality of storage units having a controller and solid-state storage memory;

a first pathway coupling the plurality of processors of the plurality of storage nodes to each other and configured to couple to a network; and

a second pathway coupling at least a first controller of a first one of the plurality of storage units and a second controller of a second one of the plurality of storage units for communication without participation by the processors of the plurality of storage nodes.

9. The storage system of claim 8, further comprising:

the first one of the plurality of storage units configured to have the first controller communicate a write request and a data shard or a parity shard to the second controller through the second pathway; and

the second one of the plurality of storage units configured to have the second controller store the data shard or the parity shard in the solid-state storage memory of the second storage unit, responsive to the write request.

10. The storage system of claim 8, further comprising:

the first one of the plurality of storage units configured to have the first controller communicate an inquiry to the second controller through the second pathway; and

the second one of the plurality of storage units configured to have the second controller communicate a reply to the first controller through the second pathway.

11. The storage system of claim 8, further comprising:

the first one of the plurality of storage units configured to have the first controller communicate a read request for data or metadata to the second controller through the second pathway; and

the second one of the plurality of storage units configured to have the second controller communicate the data or the metadata to the first controller through the second pathway.

12. The storage system of claim 8, further comprising:

one of the plurality of storage nodes configured to have a processor communicate a direction to construct data from data shards to the first controller; and

the first one of the plurality of storage units configured to have the first controller communicate at least one request for at least one data shard to the second controller through the second pathway.

13. The storage system of claim 8, further comprising:

the first and second ones of the plurality of storage units configured to have the first controller and the second controller each communicate a heartbeat through the second pathway.

14. The storage system of claim 8, further comprising:

a network switch, through which the first pathway passes; and

a PCI (peripheral component interconnect) Express bus, through which the second pathway passes.

15. A method for communication in a storage system, performed by processors in the storage system, comprising:

communicating among two or more of a plurality of processors of a plurality of storage nodes of a storage system through a first pathway; and

communicating among at least a first controller of a first one of a plurality of storage units of the storage system and a second controller of a second one of the plurality of storage units through a second pathway, wherein each of the plurality of storage units has solid-state storage memory and wherein the second pathway is a direct controller to controller coupling.

16. The method of claim 15, further comprising:

sending a write request and a data shard or a parity shard from the first controller of the first one of the plurality of storage units to the second controller of the second one of the plurality of storage units, through the second pathway; and

writing the data shard or the parity shard into the solid-state storage memory of the second one of the plurality of storage units.

17. The method of claim 15, further comprising:

sending an inquiry from the first controller of the first one of the plurality of storage units to the second controller of the second one of the plurality of storage units, through the second pathway; and

sending a reply from the second controller of the second one of the plurality of storage units to the first controller of the first one of the plurality of storage units, through the second pathway.

18. The method of claim 15, further comprising:

sending a read request for data or metadata from the first controller of the first one of the plurality of storage units to the second controller of the second one of the plurality of storage units, through the second pathway; and

sending the data or the metadata from the second controller of the second one of the plurality of storage units to the first controller for of the first one of the plurality of storage units through the second pathway.

19. The method of claim 15, further comprising:

communicating a direction to construct data from data shards from a processor of one of the plurality of storage nodes to the first controller of the first one of the plurality of storage units; and

sending a plurality of requests for a plurality of data shards from the first controller of the first one of the plurality of storage units to further controllers of further ones of the plurality of storage units through the second pathway.

20. The method of claim 15, further comprising:

communicating a heartbeat from the first controller of the first one of the plurality of storage units to the second controller of the second one of the plurality of storage units through the second pathway, and wherein the second pathway is a direct controller to controller connection.