MODULAR SWITCHED FABRIC FOR DATA STORAGE SYSTEMS

• US 20150373115A1
• Filed: 06/23/2015
• Published: 12/24/2015
• Est. Priority Date: 06/23/2014
• Status: Active Grant

• Current Assignee: Liqid Inc
• Original Assignee: Liqid Inc
• Inventors: Breakstone, Jason, Long, Christopher R., Cannata, James Scott
• Granted Patent: US 9,645,902 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/2012 and using different communication protocols
• CPC Class Codes: G06F 11/2017 where memory access, memory control or I/O control functionality is redundant redundant communication control functionality G06F11/2005; redundant storage control functionality G06F11/2089
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2046 where the redundant components share persistent storage G06F11/2043 takes precedence
• CPC Class Codes: G06F 11/2053 where persistent mass storage functionality or persistent mass storage control functionality is redundant error detection or correction in information storage based on relative movement between record carrier and transducer G11B20/18
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 13/4068 Electrical coupling
• CPC Class Codes: G06F 13/4234 being a memory bus
• CPC Class Codes: G06F 13/4282 on a serial bus, e.g. I2C bus, SPI bus on daisy chain buses G06F13/4247
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares
• CPC Class Codes: G06F 2213/0026 PCI express
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 49/356 for storage area networks
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/40 : for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Systems, methods, apparatuses, and software for data storage systems are provided herein. In one example, a data storage system is provided that includes a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations. The data storage system includes one or more processing modules comprising one or more processors communicatively coupled to the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches. The processors are configured to share a PCIe address space associated with the PCIe fabric for transfer of the storage operations to appropriate ones of the processors that manage ones of the plurality of storage drives.

Claims

1. A data storage system, comprising:

a plurality of storage drives each comprising a Peripheral Component Interconnect Express (PCIe) interface, and configured to store data and retrieve the data stored on associated storage media responsive to storage operations;

one or more processing modules comprising one or more processors communicatively coupled to the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches;

the processors configured to share a PCIe address space associated with the PCIe fabric for transfer of the storage operations to appropriate ones of the processors that manage ones of the plurality of storage drives.

2. The data storage system of claim 1, comprising:

a first of the processors configured to receive a first storage operation and identify a portion of the PCIe address space associated with the first storage operation;

when the portion of the PCIe address space is associated with the second of the processors, the first of the processors configured to transfer the first storage operation for delivery to a second of the processors over the PCIe fabric for handling of the storage operation with ones of the plurality of storage drives assigned to the second of the processors.

3. The data storage system of claim 1, comprising:

a front end data plane formed among at least a first one of the PCIe switches that communicatively couples the processors to each other; and

a back end data plane formed among at least a second one of the PCIe switches that communicatively couples the processors to the plurality of storage drives.

4. The data storage system of claim 1, comprising:

a unified data plane formed among the one or more PCIe switches that communicatively couples the processors to each other, and communicatively couples the processors to the plurality of storage drives.

5. The data storage system of claim 1, comprising:

the one or more processors communicatively coupled to network interfaces over the PCIe fabric, the network interfaces configured to receive the storage operations over associated network links and transfer the storage operations to the one or more processors over the PCIe fabric.

6. The data storage system of claim 5, comprising:

each of the network interfaces configured to be associated with a different Internet Protocol (IP) address and at least two of the network interfaces configured to be associated with a first Internet Small Computer System Interface (iSCSI) portal group;

a first processor associated with a first network interface and associated with the first iSCSI portal group configured to identify data packets received over the first network interface as comprising a storage operation associated with at least one of the plurality of storage drives managed by a second processor, and responsively transfer the first data packets for handling by the processor having a second network interface associated with the first iSCSI portal group; and

the second processor configured to handle the storage operation associated with the data packets as if the second network interface had received the data packets.

7. The data storage system of claim 1, comprising:

at least one of the PCIe switches configured to interconnect processing modules of the data storage system with another data storage system over at least one external PCIe link.

8. The data storage system of claim 7, comprising:

the at least one external PCIe links comprising PCIe signaling carried over Serial Attached Small Computer Serial Interconnect (SAS) connectors and cabling to interconnect the data storage system to the other data storage system.

9. The data storage system of claim 1, comprising:

at least one of the processors configured to present an associated one or more storage drives as at least one NVM Express (NVMe) virtual logical unit number (VLUN) over a PCIe interface that communicatively couples the at least one of the processors and a computing system external to the data storage system;

the at least one of the processors configured to receive NVMe storage transactions transferred by the computing system over the PCIe interface and handle the NVMe storage transactions for storage of data on the associated one or more storage drives.

10. A method of operating a data storage system, the method comprising:

in a plurality of storage drives, storing data and retrieving the data stored on associated storage media responsive to storage operations received over a Peripheral Component Interconnect Express (PCIe) interface;

in one or more processing modules comprising one or more processors communicatively coupled to the plurality of storage drives over a PCIe fabric comprised of one or more PCIe switches, sharing a PCIe address space associated with the PCIe fabric for transfer of the storage operations to appropriate ones of the processors that manage ones of the plurality of storage drives.

11. The method of claim 10, further comprising:

in a first of the processors, receiving a first storage operation and identifying a portion of the PCIe address space associated with the first storage operation;

when the portion of the PCIe address space is associated with the second of the processors, in the first of the processors, transferring the first storage operation for delivery to a second of the processors over the PCIe fabric for handling of the storage operation with ones of the plurality of storage drives assigned to the second of the processors.

12. The method of claim 10, wherein a front end data plane is formed among at least a first one of the PCIe switches that communicatively couples the processors to each other;

and wherein a back end data plane is formed among at least a second one of the PCIe switches that communicatively couples the processors to the plurality of storage drives.

13. The method of claim 10, wherein a unified data plane is formed among the one or more PCIe switches that communicatively couples the processors to each other, and communicatively couples the processors to the plurality of storage drives.

14. The method of claim 10, further comprising:

communicatively coupling the one or more processors to network interfaces over the PCIe fabric;

in the network interfaces, receiving the storage operations over associated network links and transferring the storage operations to the one or more processors over the PCIe fabric.

15. The method of claim 14, further comprising:

associating each of the network interfaces with a different Internet Protocol (IP) address and associating at least two of the network interfaces with a first Internet Small Computer System Interface (iSCSI) portal group;

in a first processor associated with a first network interface and associated with the first iSCSI portal group, identifying data packets received over the first network interface as comprising a storage operation associated with at least one of the plurality of storage drives managed by a second processor, and responsively transferring the first data packets for handling by the processor having a second network interface associated with the first iSCSI portal group; and

in the second processor handling the storage operation associated with the data packets as if the second network interface had received the data packets.

16. The method of claim 10, further comprising:

in at least one of the PCIe switches, interconnecting the processing modules of the data storage system with processing modules of another data storage system over at least one external PCIe link, wherein the at least one external PCIe links comprises PCIe signaling carried over Serial Attached Small Computer Serial Interconnect (SAS) connectors and cabling to interconnect the data storage system to the other data storage system.

17. The method of claim 10, further comprising:

in at least one of the processors, presenting an associated one or more storage drives as at least one NVM Express (NVMe) virtual logical unit number (VLUN) over a PCIe interface that communicatively couples the at least one of the processors and a computing system external to the data storage system;

in the at least one of the processors receiving NVMe storage transactions transferred by the computing system over the PCIe interface and handling the NVMe storage transactions for storage of data on the associated one or more storage drives.

18. A modular data storage system, comprising:

a plurality of storage modules each comprising a plurality of solid state drives and configured to communicate over a switched Peripheral Component Interconnect Express (PCIe) fabric;

a plurality of processing modules each comprising one or more processors that are configured to manage storage operations for a designated subset of the plurality of storage modules over the switched PCIe fabric;

a plurality of network interfaces individually associated with ones of the processing modules and configured to communicate with the associated ones of the processing modules over the switched PCIe fabric;

one or more PCIe switches configured to establish the switched PCIe fabric over a plurality of PCIe links to interconnect the storage modules to the processing modules and interconnect the processing modules to the network interfaces;

the processors configured to share a PCIe address space associated with the switched PCIe fabric for transfer of received storage operations to appropriate ones of the processors that manage ones of the plurality of storage drives.

19. The modular data storage system of claim 18, comprising:

each of the network interfaces configured to be associated with a different Internet Protocol (IP) address and at least two of the network interfaces configured to be associated with a first Internet Small Computer System Interface (iSCSI) portal group;

a first processor associated with a first network interface and associated with the first iSCSI portal group configured to identify data packets received over the first network interface as comprising a storage operation associated with at least one of the plurality of storage drives managed by a second processor, and responsively transfer the first data packets for handling by the processor having a second network interface associated with the first iSCSI portal group; and

the second processor configured to handle the storage operation associated with the data packets as if the second network interface had received the data packets.

20. The modular data storage system of claim 18, comprising:

at least one of the processors configured to present an associated one or more storage drives as at least one NVM Express (NVMe) virtual logical unit number (VLUN) over a PCIe interface that communicatively couples the at least one of the processors and a computing system external to the data storage system;

the at least one of the processors configured to receive NVMe storage transactions transferred by the computing system over the PCIe interface and handle the NVMe storage transactions for storage of data on the associated one or more storage drives.