System apparatus and method for storage device controller-based message passing having effective data channel bandwidth and controller cache memory increase

• US 6,792,505 B2
• Filed: 04/16/2001
• Issued: 09/14/2004
• Est. Priority Date: 04/16/2001
• Status: Expired due to Term

• Current Assignee: Xyratex Technology Limited (Seagate Technology Holdings Plc)
• Original Assignee: International Business Machines Corporation
• Inventors: Skazinski, Joseph G., Otterness, Noel S.
• Primary Examiner(s): Sparks, Donald
• Assistant Examiner(s): Truong, Bao Q.
• Application Number:
  US09/836,806
• Publication Number:
  US 20020152355A1
• Time in Patent Office: 1,247 Days
• Field of Search: 711/114, 711/48, 711/153, 710/100, 710/308, 710/38, 710/311, 710/316, 709/212, 709/216, 709/239
• US Class Current: 711/114
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

Controller for coupling data between a data storage system and a host includes a first processor and a first RAM coupled to the first processor; a first auxiliary processor including a first memory controller and a first cache coupled to the first memory controller, the first memory controller including first interface for coupling with second auxiliary processor including second memory controller and associated second cache and second interface for coupling with first auxiliary processor, first memory controller including logic for treating the caches as single memory; a bus coupling first primary processor and first auxiliary processor; and interconnection channel separate from the bus coupling first interface of first memory controlled and second interface of second memory controller. Interconnection may be an out-of-band channel permitting device-to-device sharing of associated cache memories without requiring data transfer over the bus. Method and computer program product are also provided.

Claims

1. A controller device for coupling data between a data storage system including a plurality of disk arrays and a host computer, said controller device comprising:

a first prima processor and a first random access memory coupled to said first primary processor;

a first auxiliary processor including a first memory controller and a first memory coupled to said first memory controller, said first memory controller including a first interface for coupling with a second auxiliary processor including a second memory controller and an associated second memory and a second interface for coupling with said first auxiliary processor, said first memory controller including control logic for treating said first memory and said second memory as a single memory complex;

at least one communications bus for communication with the plurality of disk arrays coupling said first primary process and said first auxiliary processor; and

an interconnection channel separate from said communications bus extending between and coupling said first interface of said first memory controller and said second interface of said second memory controller.

2. The controller device in claim 1, wherein said first auxiliary processor further comprises an exclusive-OR engine.

3. The controller device in claim 2, wherein said exclusive-OR engine executes instructions performing RAID Level 5 parity calculations.

4. The controller device in claim 1, wherein said at least one communications bus comprises at least on PCI bus.

5. The controller device in claim 1, wherein said at least one communications bus comprises first and second PCI buses, said primary processor and said auxiliary processor each being coupled to each of said first and said second PCI busses.

6. The controller device in claim 5, wherein said interconnection channel comprises an out-of-band interconnect physical channel and protocol that permit device-to-device sharing their associated first and second memories without requiring data transfer over the first or second PCI busses.

7. The controller device in claim 6, wherein said interconnection physical channel is selected from the set of physical channels consisting of a serial channel, an SCI channel, a fire wire channel, a SCSI channel, an Ethernet channel, a Fibre channel, Infiniband Connection, and combinations thereof.

8. The controller device in claim 2, wherein said XOR engine comprises a programmable script-based engine allowing multiple different memory locations to be XOR'"'"'d together and the results of the XOR operation to be stored in separate memories.

9. The controller device in claim 5, wherein said first memory controller allows data to be brought in from either of the two PCI buses and stored in the first memory associated with said first memory controller.

10. The controller device in claim 1, wherein said first memory controller has primary responsibility for a write operation and said second memory controller is a partner controller to said first memory controller, and wherein host write data is mirrored to said partner controller over said interconnection in connection with a write back I/O operation for said first memory controller.

11. The controller device in claim 1, wherein said primary processor, and said first and second auxiliary processors are disposed on a common substrate.

12. The controller device in claim 11, wherein said common substrate comprises a PC board.

13. The controller device in claim 1, wherein said primary processor and said first auxiliary processor are disposed on a first common substrate, and said second auxiliary processor is an external auxiliary processor disposed on a second substrate different from said first substrate.

14. The controller device in claim 1, wherein said first and said second memory operate as first and second cache memories.

15. A controller device for coupling data between a data storage system including a plurality of disk arrays and a host computer, said controller device comprising:

a first prima processor and a first random access memory coupled to said first processor;

a first auxiliary processor including a memory controller and an XOR processing engine executing instructions for RAID parity calculations, and a first cache memory coupled to said first memory controller, said first memory controller including a first interface for coupling with a second auxiliary processor including a second memory controller and an associated second cache memory and second interface for coupling with said first auxiliary processor, said first memory controller including control logic for treating said first cache memory and a second cache memory associated with and coupled to said second memory controller as a single memory complex;

at least one communications PCI bus for communication with the plurality of disk arrays coupling said first primary processor and said first auxiliary processor; and

an interconnection channel separate from said communications bus extending between and coupling said first interface of said first memory controller and said second interface of said second memory controller, said interconnection channel comprising an out-of-band interconnect physical channel protocol that permit device-to-device sharing of their associated first and second cache memories without requiring data transfer over said PCI bus.

16. The controller device in claim 15, wherein said XOR engine comprises a programmable script-based engine allowing multiple different memory locations to be XOR'"'"'d together and the results of the XOR operation to be stored in separate memories.

17. The controller device in claim 16, wherein said first memory controller allows data to be brought in ver said PCI bus and stored in the first cache memory associated with said first memory controller.

18. The controller device in claim 15, wherein said first auxiliary controller is operative with a device controller has primary responsibility for a write operation and said second auxiliary controller is operative in a second controller device that is a partner controller device to said first controller device, and in connection with a write back I/O operation for said first device controller, host write data is mirrored to said partner controller over said interconnection.

19. In a computer system having a plurality of host computers, and a data storage system configured at east in part as a RAID and shared by said plurality of host computers and controlled by a plurality of controllers, each controller including at least one processor and a memory associated with and coupled to said processor, said processors being coupled by a system bus and by an interconnect different from said system bus, a method of performing a write operation to said storage system comprising:

receiving a host write command including a host write data;

identifying the memory or memories wherein are stored XOR operation data required to perform a RAID XOR operation including memories storing relevant prior write data for a RAID stripe;

identifying a single memory from amongst all available memories to gather all of said XOR operation data;

transferring all of said XOR operation data from memories other than said identified single memory to gather all of said XOR operation data in said identified memory over said direct interconnection different from said system bus;

performing the XOR operation using said gathered XOR operation data; and

directing the write data and the parity data from said XOR operation to the RAID storage system.

20. The method in claim 19, wherein said single memory is identified from amongst said memories already storing some XOR operation data.

21. In data storage system storing user data and parity data associated with said user data and controlled by a plurality of controllers, each of said controllers including at least one processor and an associated memory, said processors being coupled by a bus and by an interconnect different from said bus, a method of performing a write operation to said storage system comprising:

receiving a write command;

identifying memories wherein are stored parity operation data required to perform a parity operation in connection with performing said write command, said parity operation data including any relevant prior write data;

identifying a memory from amongst said identified memories to gather all of said parity operation data;

transferring all of said parity operation data to said identified memory over a direct interconnection different from a system bus;

performing the parity operation using a single processor and its associated memory; and

directing the write data and the parity data from said parity operation to the storage system.

22. The method in claim 21, wherein said write command includes write data.

23. The method in claim 21, wherein said parity operation data further includes any relevant prior write data.

24. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism, comprising:

a program module that directs a computer system having a plurality of hosts, and a data storage system configured as RAID shared by said plurality of hosts and controlled by a plurality of controllers, to function in a specified manner, each controller including at least one processor and a memory associated with said processor, the program module including instructions for;

receiving a host write command including a host write data;

identifying the memory or memories wherein are stored all XOR operation data required to perform a RAID XOR operation, including any relevant prior write data for a RAID stripe;

identifying a single memory from amongst said identified memory or memories to gather all of said XOR operation data;

transferring all of said data from memories other than said identified single memory to said identified memory over a direct interconnection different from a system bus;

performing the XOR operation; and

directing the write data and the parity data from said XOR operation to the RAID storage.

25. A computer program product for use in conjunction with a data storage system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism, comprising:

a program module that directs a data storage system storing user data and parity data associated with said user data and controlled by a plurality of controllers each including at least one processor and an associated memory, and being coupled by a bus and by an interconnect different from said bus, to function in a specified manner, the program module including instructions for;

receiving a write command;

identifying memories wherein are stored parity operation data required to perform a parity operation in connection with performing said write command, said parity operation data including any relevant prior write data;

identifying a memory from amongst said identified memories to gather all of said parity operation data;

transferring all of said parity operation data to said identified memory over a direct interconnection different from a system bus;

performing the parity operation using a single processor and its associated memory; and

directing the write data and the parity data from said parity operation to the storage system.

26. The computer program product in claim 25, wherein said write command includes write data.

27. The method in claim 25, wherein said parity operation data further includes any relevant prior write data.

28. An apparatus for communicating a message between first and second devices, each said first and second device having respective first and second processors, said apparatus comprising:

a communication bus coupled to the first and second processors for communication with a plurality of disk array;

a first memory controller separate from said first processor and a first memory coupled to and controlled by said first memory controller;

a second memory controller separate from said second processor and a second memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said communication bus, coupling said first and said second memory controllers for out-of-band communication between said first and second devices; and

a communication path selector directing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules.

29. The apparatus in claim 28, wherein said devices comprise at least two storage device controllers.

30. The apparatus in claim 29, wherein said message comprises controller configuration information.

31. The apparatus in claim 29, wherein said message comprises cache information.

32. The apparatus in claim 29 wherein said message comprises information facilitating maintaining coherency between caches within said controllers.

33. The apparatus in claim 28, wherein said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.

34. The apparatus in claim 33, wherein said computer program executes within at least one of said first and second processors.

37. The apparatus in claim 28, wherein:

said devices comprise at least two storage device controllers;

said message comprises information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, and combinations thereof; and

said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.

38. The apparatus in claim 37, wherein said computer program executes within at least one of said first and second processors.

35. An apparatus for communicating a message between first and second devices, each said first and second device having respective first and second processors coupled for in-band communication by a communication bus, said apparatus comprising:

a first memory controller separate from said first processor and a first memory coupled to and controlled by said first memory controller;

a second memory controller separate from said second processor and a second memory coupled to and controlled by said second memory controller;

a communication interconnect, separate from said communication bus, coupling said first and said second memory controllers for out-of-band communication between said first and second devices;

a communication path selector directing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules, and wherein said computer program executes within at tleast one of said first and second memory controllers.

36. An apparatus for communicating a message between first and second devices, each said first and second device having respective first and second processors coupled for in-band communication by a communication bus, said apparatus comprising;

a first memory controller separate from said first processor and a first memory coupled to and controlled by said firs memory controller;

a second memory controller separate from said second processor and a second memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said communication bus, coupling said first and said second memory controllers for out-of-band communication between said first and second devices;

a communication path selector directing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules, and wherein said communication path selector comprises a path selector procedure implemented as first and second computer programs having instructions executing within at least one of said first and second processors and within at least one of said first and second memory controllers.

39. An apparatus for communicating a message, said message comprising information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, and combinations thereof, between first and second devices, each said first and second device having respective first and second processors coupled for in-band communication by a communication bus, and each of said devices comprising at least two storage device controllers, said apparatus comprising:

a first memory controller separate from said first processor and a first memory coupled to and controlled by said first memory controller;

a second memory controller separate from said second processor and a second memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said communication bus, coupling said first and said second memory controllers for out-of-band communication between said first and second devices; and

a communication path selector directing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules, the communication path selector comprising a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices, wherein said computer program executes within at least one of said first and second memory controllers.

40. An apparatus for communicating a message, said message comprising information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, and combinations thereof, between first and second devices, each said first and second device having respective first and second processors coupled for in-band communication by a communication bus, and each of said devices comprising at least two storage device controllers, said apparatus comprising:

a first memory controller separate from said first processor and a first memory coupled to and controlled by said first memory controller;

a second memory controller separate from said second processor and a second memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said communication bus, coupling said first and said second memory controllers for out-of-band communication between said first and second devices; and

a communication path selector directing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined select ion rules;

wherein said communication path selector comprises a path selector procedure implemented as first and second computer programs having instructions executing within at least one of said first and second processors and within at least one of said first and second memory controllers.

41. An apparatus for communicating a message between first and second storage device controllers, each said first and second storage device controllers having respective first and second processors, said apparatus comprising:

a PCI-based communication bus coupled to the first and second processors for communication with a plurality of disk array;

a first memory controller and first RAID parity operation processor separate from said first processor and a first cache memory coupled to and controlled by said first memory controller;

a second memory controller and second RAID parity operation processor separate from said second processor and a second cache memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said PCI-based communication bus, coupling said first and said second memory controllers and parity operation processors for out-of-band communication of said message between said first and second devices;

said message comprises information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, an combinations thereof; and

a communication path selector routing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules, said selection rules including a load-leveling rule taking into account bus loading and interconnect loading and attempting to achieve a desired loading ratio.

42. The apparatus in claim 41, wherein said message comprises controller configuration information.

43. The apparatus in claim 41, wherein said message comprises cache information.

44. The apparatus in claim 41, wherein said message comprises information facilitating maintaining coherency between caches within said controllers.

45. The apparatus in claim 41, wherein said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.

46. An apparatus for communicating a message between first and second RAID storage device controllers, each said first and second storage device controllers having respective first and second processors coupled for in-band communication by a PCI-based communication bus, said apparatus comprising:

a first memory controller and first RAID parity operation processor separate from said first processor and a first cache memory coupled to and controlled by said first memory controller;

a second memory controller and second RAID parity operation processor separate from said second processor and a second cache memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said PCI-based communication bus, coupling said first and said second memory controllers and parity operation processors for out-of-band communication of said message between said first and second devices;

said message comprises information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, and combinations thereof; and

a communication path selector routing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules, said selection rules including a load-leveling rule taking into account bus loading and interconnect loading and attempting to achieve a desired loading ratio, wherein said desired loading ratio is about a 50%—

50% loading ratio.

47. A system for communication between a first memory controller and a second memory controller, the system for use with a plurality of hosts and a plurality of disk arrays, the system comprising:

a first host bus coupled to the first memory controller and operable to communicate with a first host of the plurality of hosts;

a first disk bus coupled to the first memory controller and operable to communicate with at least a first disk array of the plurality of disk arrays;

a second host bus coupled to the second memory controller and operable to communicate with a second host of the plurality of hosts;

a second disk bus coupled to the second memory controller and operable to communicate with at least a second disk array of the plurality of disk arrays;

a first interconnect coupling said first disk bus and said second disk bus; and

an out-of-band interconnect, separate from said first interconnect, coupled between said first and second memory controllers.

48. A system according to claim 47, further comprising a second interconnect, separate from said out-of-band interconnect, coupling said first host bus and said second host bus.

49. A system according to claim 47, wherein the first interconnect comprises a PCI bridge.

50. A system according to claim 47, wherein the out-of-band interconnect comprises an interconnect selected from the group of interconnects consisting of a serial channel, an SCI channel, a fire wire channel, a SCSI channel, an Ethernet channel, a Fibre channel, and an Infiniband Connection.

51. A system according to claim 47, wherein the first and second disk buses are each operable to communicate with a first and second plurality of disk arrays.