Controller communications over an always-on controller interconnect

• US 7,480,815 B2
• Filed: 06/10/2005
• Issued: 01/20/2009
• Est. Priority Date: 05/14/2002
• Status: Active Grant

• Current Assignee: Valtrus Innovations Limited (f/k/a Dolya Holdco 9 Limited) (Key Patent Innovations Limited)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Rust, Robert A., Van De Graaff, Tammy T., Oldfield, Barry J.
• Primary Examiner(s): Maskulinski; Michael C
• Application Number:
  US11/150,796
• Publication Number:
  US 20050246579A1
• Time in Patent Office: 1,320 Days
• Field of Search: 714/4, 714/9, 714/11
• US Class Current: 714/4.12
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/201 between storage system components
• CPC Class Codes: G06F 11/2056 by mirroring
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2092 Techniques of failing over between control units
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0673 Single storage device

Abstract

A controller interconnect structure within a RAID disk array enables continuous low latency/high bandwidth communications between a plurality of controller pairs within the array. Mirror buses carry high speed mirror traffic between mirrored controllers performing mirrored memory operations. Loop buses carry inter-processor communications and other traffic between controller pairs coupled together in a controller loop. Benefits of the interconnect structure include an ability to support continued controller communications and online disk array operations under various failure and repair conditions that might otherwise render a disk array inoperable. In addition, the controller interconnect structure provides for easy expansion of the number of controllers within disk arrays as arrays continue to be scaled up in size to meet increasing storage demands from user host systems.

Claims

1. A disk array comprising:

at least two controller pairs, each controller pair having a first controller coupled to a second controller through a mirror bus, wherein each bus is divided into a first half and a second half;

a controller loop coupling all controllers through a plurality of loop buses;

a first interconnect to which each controller is operatively coupled, the first interconnect carrying the first half of each bus; and

a second interconnect to which each controller is operatively coupled, the second interconnect carrying the second half of each bus.

2. A disk array as recited in claim 1, wherein each controller in the controller loop is coupled to a first logically adjacent controller through a first loop bus and to a second logically adjacent controller through a second loop bus.

3. A disk array as recited in claim 1, wherein the controller loop further comprises:

a first controller row having all first controllers coupled as a row of controllers; and

a second controller row having all second controllers coupled as a row of controllers; and

wherein a beginning controller in the first controller row is coupled to an ending controller in the second controller row and an ending controller in the first controller row is coupled to a beginning controller in the second controller row.

4. A disk array as recited in claim 1, wherein the first interconnect and the second interconnect are selected from a group of interconnects comprising:

a back plane; and

a cable.

5. A disk array as recited in claim 1, wherein each controller comprises routing logic configured to route data along the controller loop.

6. A disk array as recited in claim 5, wherein the routing logic comprises a programmable routing register configured to contain routing information usable by the routing logic.

7. A disk array as recited in claim 1, wherein each controller comprises a hardware circuit configured to detect a failure in the controller loop and to reroute data to avoid the failure.

8. A disk array as recited in claim 7, wherein the hardware circuit is further configured to modify contents of a routing register so that data is initially routed to avoid the failure.

9. A controller interconnect structure in a disk array comprising:

a plurality of controllers;

a plurality of communications buses operatively coupling the plurality of controllers to one another, each communications bus partitioned into a first half and a second half;

a first interconnect to which each of the plurality of controllers is coupled, the first interconnect conveying the first half of the plurality of communications buses; and

a second interconnect to which each of the plurality of controllers is coupled, the second interconnect conveying the second half of the plurality of communications buses.

10. A controller interconnect structure as recited in claim 9, wherein the plurality of controllers comprises a plurality of controller pairs, each controller pair having a first controller and a second controller.

11. A controller interconnect structure as recited in claim 10, wherein the plurality of communications buses comprises:

between each controller pair, a mirror bus operatively coupling the first controller to the second controller; and

a loop bus operatively coupling each controller with two logically adjacent controllers such that all controllers form part of a continuous controller loop.

12. A controller interconnect structure as recited in claim 9, wherein each controller further comprises routing logic configured to route data along the controller loop to a destination controller pair based on an address header and a routing instruction.

13. A controller interconnect structure as recited in claim 12, wherein the routing logic comprises a programmable routing register configured to contain the routing instruction, the routing logic further configured to access the programmable routing register and route data along the controller loop to a destination controller pair based on the routing instruction.

14. A controller interconnect structure as recited in claim 9, wherein each controller comprises a hardware circuit configured to detect a failure in the interconnect structure and to reroute data over the interconnect structure to avoid the failure.

15. A controller interconnect structure as recited in claim 14, wherein the hardware circuit is further configured to modify contents of a routing register so that data is initially routed over the interconnect structure to avoid the failure.

16. A controller interconnect structure as recited in claim 9, wherein the first interconnect and the second interconnect are selected from a group of interconnects comprising:

a back plane; and

a cable.

17. A processor-readable medium comprising processor-executable instructions configured for:

determining a destination controller pair for a data packet based on a host address for the data packet and an array mapping of the host address to an array address;

initially sending the data packet over a controller loop toward the destination controller pair in a first direction determined by a data header and a routing instruction;

detecting a failure in the controller loop;

rerouting the data packet in a second direction toward the destination controller pair;

based on the failure, reprogramming a routing register with new routing information;

based on the new routing information, initially sending additional data packets in a direction over the controller loop that avoids the failure;

sharing failure information with all controllers; and

based on the failure information, reprogramming routing registers on each controller with new routing information.

18. A processor-readable medium as recited in claim 17, wherein the determining further comprises attaching the data header to the data packet.

19. A processor-readable medium comprising processor-executable instructions configured for:

receiving at a first controller in a first controller level, data that is destined for a second level controller pair located in a second controller level;

sending the data over a first level controller loop to a first level controller pair that corresponds to the second level controller pair; and

further sending the data from the first level controller pair to the second level controller pair via a loop bus that couples the first controller level to the second controller level.

20. A processor-readable medium as recited in claim 19, comprising further processor-executable instructions configured for:

detecting a failure in a controller loop; and

rerouting data traveling over the controller loop from a first direction to a second direction to avoid the failure.

21. A processor-readable medium as recited in claim 20, comprising further processor-executable instructions configured for:

based on the failure, programming routing registers on all controllers in the controller loop with routing information; and

based on the routing information, initially sending additional data in a direction over the controller loop that avoids the failure.

22. A processor-readable medium as recited in claim 20, comprising further processor-executable instructions configured for:

sharing failure information with all controllers; and

based on the failure information, reprogramming routing registers on each controller with new routing information.

23. A processor-readable medium comprising processor-executable instructions configured for:

receiving data at a first controller in a first controller level;

sending the data from the first controller to a second level controller pair via a loop bus that couples the first controller level to a second controller level; and

further sending the data over a second level controller loop to a destination controller pair in the second controller level.

24. A method of routing data between controller pairs in a multi-controller disk array comprising:

receiving at a first controller in a first controller level, data that is destined for a second level controller pair located in a second controller level;

sending the data over a first level controller loop to a first level controller pair that corresponds to the second level controller pair; and

further sending the data from the first level controller pair to the second level controller pair via a loop bus that couples the first controller level to the second controller level.

25. A method of routing data between controller pairs in a multi-controller disk array comprising:

receiving data at a first controller in a first controller level;

sending the data from the first controller to a second level controller pair via a loop bus that couples the first controller level to a second controller level; and

further sending the data over a second level controller loop to a destination controller pair in the second controller level.