Storage array having multiple controllers

• US 9,720,598 B2
• Filed: 02/12/2016
• Issued: 08/01/2017
• Est. Priority Date: 11/15/2011
• Status: Active Grant

• Current Assignee: Pavilion Data Systems, Inc.
• Original Assignee: Pavilion Data Systems, Inc.
• Inventors: Malwankar, Kiron Balkrishna
• Primary Examiner(s): Rones, Charles
• Assistant Examiner(s): DOAN, HAN V
• Application Number:
  US15/042,849
• Publication Number:
  US 20160162189A1
• Time in Patent Office: 536 Days
• CPC Class Codes: G06F 12/10 Address translation
• CPC Class Codes: G06F 13/1668 Details of memory controller
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 2212/65 Details of virtual memory and virtual address translation
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A storage system comprises a storage array comprising a plurality of solid state storage devices (SSDs), a first processor comprising a first root complex of the storage system, a plurality of controller devices, and a first switch to interconnect the plurality of SSDs, the first processor and the plurality of controller devices. A first controller device of the plurality of controller devices is to connect the storage system to one or more remote servers. The first controller device is further to receive a first request from a first server of the one or more remote servers and determine whether the first request is a data request or a control request. The first controller device is further to send a first message to a first SSD of the plurality of SSDs via the first switch, bypassing the first processor, responsive to a determination that the first request is a data request.

Claims

1. A storage system comprising:

a storage array comprising a plurality of peripheral component interconnect express (PCIe) type solid state storage devices (SSDs);

a plurality of controller devices, each of the plurality of controller devices comprising a processor;

a first processor comprising a first PCIe type root complex of the storage system, the first processor to;

determine a plurality of PCIe type endpoints of the storage system, the plurality of PCIe type endpoints comprising the plurality of controller devices and the plurality of PCIe type SSDs;

allocate memory addresses to each of the plurality of PCIe type endpoints;

generate a mapping table comprising one or more of the memory addresses allocated to one or more of the plurality of PCIe type endpoints; and

send the mapping table to at least a first controller device of the plurality of controller devices; and

a first PCIe type switch to interconnect the plurality of PCIe type SSDs, the first processor and the plurality of controller devices using a PCIe type interface, wherein the plurality of PCIe type SSDs and the plurality of controller devices each comprise a PCIe type endpoint;

wherein the first controller device is to;

connect the storage system to one or more remote servers;

receive a first request from a first server of the one or more remote servers;

determine whether the first request is a data request or a control request;

responsive to a determination that the first request is the data request associated with a data path, send a first message from a first PCIe type endpoint of the first controller device to a second PCIe type endpoint of a first SSD of the plurality of PCIe type SSDs via the first PCIe type switch using a peer-to-peer communication mechanism, bypassing the first processor and the first PCIe type root complex of the storage system; and

responsive to a determination that the first request is the control request associated with a control path, send a second message from the first PCIe type endpoint of the first controller device to the first PCIe type root complex.

2. The storage system of claim 1, further comprising:

a second processor comprising a second root complex for the storage system; and

a second switch to interconnect the plurality of PCIe type SSDs, the second processor and the plurality of controller devices.

3. The storage system of claim 1, wherein the first message is sent to the second PCIe type endpoint of the first SSD based on the mapping table.

4. The storage system of claim 1, further comprising:

a second controller device of the plurality of controller devices to;

connect the storage system to one or more additional remote servers;

receive a second request from a second server of the one or more additional remote servers;

determine whether the second request is a data request or a control request; and

send a second message to the first SSD of the plurality of PCIe type SSDs via the first switch, bypassing the first processor, responsive to a determination that the second request is a data request.

5. The storage system of claim 1, wherein the first processor is to:

determine a first paired queue associated with the first SSD, the first paired queue comprising a first submission queue and a first completion queue, wherein the first paired queue is to reside in a memory of the first controller device; and

determine a second paired queue associated with a second SSD of the plurality of PCIe type SSDs, the second paired queue comprising a second submission queue and a second completion queue, wherein the second paired queue is to reside in the memory of the first controller device.

6. The storage system of claim 1, wherein the first controller device is further to:

receive a second request from a second server of the one or more remote servers;

determine whether the second request is a data request or a control request; and

forward the second request to the first processor via the first switch responsive to a determination that the second request is a control request.

7. The storage system of claim 1, wherein the plurality of controller devices comprises a first number of controller devices, the plurality of PCIe type SSDs comprises a second number of SSDs, and the first number of controller devices is different from the second number of SSDs.

8. The storage system of claim 1, wherein the first PCIe type root complex is a master and the plurality of PCIe type SSDs and the plurality of controller devices are peers.

9. The storage system of claim 1, wherein the first controller device is to:

expose a first virtual drive to the first server, wherein the first request comprises a virtual address of the first virtual drive; and

translate the virtual address into a physical address of at least one SSD of the plurality of PCIe type SSDs using a translation table.

10. A method comprising:

connecting, by a first controller device of a storage system, the storage system to one or more remote servers, wherein the storage system comprises

     1) a plurality of controller devices comprising the first controller device, wherein the first controller device comprises a first processor,

     2) a second processor comprising a first peripheral component interconnect express (PCIe) type root complex of the storage system,

     3) a storage array comprising a plurality of PCIe type solid state storage devices (SSDs), and

     4) a first PCIe type switch to interconnect the plurality of PCIe type SSDs, the second processor and the plurality of controller devices using a PCIe type interface, wherein the plurality of PCIe type SSDs and the plurality of controller devices each comprise a PCIe type endpoint;

determining, by the second processor, a plurality of PCIe type endpoints of the storage system, the plurality of PCIe type endpoints comprising the plurality of controller devices and the plurality of PCIe type SSDs;

allocating, by the second processor, memory addresses to each of the plurality of PCIe type endpoints;

generating, by the second processor, a mapping table comprising one or more of the memory addresses allocated to one or more of the plurality of PCIe type endpoints;

sending, by the second processor, the mapping table to at least the first controller device via the first PCIe type switch;

receiving, by the first controller device, a first request from a first server of the one or more remote servers;

determining, by the first controller device, whether the first request is a data request or a control request;

responsive to a determination that the first request is the data request associated with a data path, sending a first message from a first PCIe type endpoint of the first controller device to a second PCIe type endpoint of a first SSD of the plurality of PCIe type SSDs via the first PCIe type switch using a peer-to-peer communication mechanism, bypassing the second processor; and

responsive to a determination that the first request is the control request associated with a control path, sending a second message from the first PCIe type endpoint of the first controller device to the first PCIe type root complex.

11. The method of claim 10, wherein the storage system further comprises a third processor comprising a second root complex and a second switch to interconnect the plurality of SSDs, the third processor and the plurality of controller devices, the method further comprising:

determining, by the third processor, the plurality of PCIe type endpoints of the storage system;

allocating, by the third processor, additional memory addresses to each of the plurality of PCIe type endpoints;

generating, by the third processor, an additional mapping table comprising one or more of the additional memory addresses allocated to one or more of the plurality of PCIe type endpoints; and

sending, by the third processor, the additional mapping table to at least the first controller device via the second switch.

12. The method of claim 10, wherein sending the first message to the second PCIe type endpoint of the first SSD comprises sending the first message based on the mapping table.

13. The method of claim 10, further comprising:

connecting, by a second controller device of the plurality of controller devices, the storage system to one or more additional remote servers;

receiving, by the second controller device, a second request from a second server of the one or more additional remote servers;

determining, by the second controller device, whether the second request is a data request or a control request; and

sending, by the second controller device, a second message to the first SSD of the plurality of PCIe type SSDs via the first switch, bypassing the second processor, responsive to a determination that the second request is a data request.

14. The method of claim 10, further comprising:

determining, by the second processor, a first paired queue associated with the first SSD, the first paired queue comprising a first submission queue and a first completion queue, wherein the first paired queue is to reside in a memory of the first controller device; and

determining, by the second processor, a second paired queue associated with a second SSD of the plurality of PCIe type SSDs, the second paired queue comprising a second submission queue and a second completion queue, wherein the second paired queue is to reside in the memory of the first controller device.

15. The method of claim 10, further comprising:

receiving, by the first controller device, a second request from a second server of the one or more remote servers;

determining, by the first controller device, whether the second request is a data request or a control request; and

forwarding, by the first controller device, the second request to the second processor via the first PCIe type switch responsive to a determination that the second request is a control request.

16. The method of claim 10, wherein the first PCIe type root complex is a master and the plurality of PCIe type SSDs and the plurality of controller devices are peers.

17. The method of claim 10, further comprising:

exposing, by the first controller device, a first virtual drive to the first server, wherein the first request comprises a virtual address of the first virtual drive; and

translating, by the first controller device, the virtual address into a physical address of at least one SSD of the plurality of PCIe type SSDs using a translation table.

18. A storage system comprising:

a storage array comprising a plurality of peripheral component interconnect express (PCIe) type solid state storage devices (SSDs);

a plurality of controller devices, each of the plurality of controller devices to connect the storage system to one or more remote servers;

a first processor comprising a first PCIe type root complex of the storage system;

a first PCIe type switch to interconnect the plurality of PCIe type SSDs, the first processor and the plurality of controller devices using a PCIe type interface, wherein the plurality of PCIe type SSDs and the plurality of controller devices each comprise a PCIe type endpoint and are peers of the storage system;

a second processor comprising a second PCIe type root complex that acts as a second master of the storage system;

a second PCIe type switch to interconnect the plurality of PCIe type SSDs, the second processor and the plurality of controller devices;

wherein the first processor is to;

determine a plurality of PCIe type endpoints of the storage system, the plurality of PCIe type endpoints comprising the plurality of controller devices and the plurality of PCIe type SSDs;

allocate memory addresses to each of the plurality of PCIe type endpoints;

generate a mapping table comprising one or more of the memory addresses allocated to one or more of the plurality of PCIe type endpoints; and

send the mapping table to at least a first controller device of the plurality of controller devices via the first PCIe type switch; and

wherein the first controller device is to;

receive a first request from a first server of the one or more remote servers;

determine whether the first request is a data request or a control request;

responsive to a determination that the first request is the data request associated with a data path, send a first message from a first PCIe type endpoint of the first controller device to a second PCIe type endpoint of a first SSD of the plurality of PCIe type SSDs via the first PCIe type switch using a peer-to-peer communication mechanism, bypassing the first processor; and

responsive to a determination that the first request is the control request associated with a control path, send a second message from the first PCIe type endpoint of the first controller device to the first PCIe type root complex.