System apparatus and method for storage device controller-based message passing having effective data channel bandwidth and controller cache memory increase

• US 20020152355A1
• Filed: 04/16/2001
• Published: 10/17/2002
• Est. Priority Date: 04/16/2001
• Status: Active Grant

• Current Assignee: Xyratex Technology Limited (Seagate Technology Holdings Plc)
• Original Assignee: International Business Machines Corporation
• Inventors: Skazinski, Joseph G., Otterness, Noel S.
• Granted Patent: US 6,792,505 B2
• Time in Patent Office: Days
• US Class Current: 711/114
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

Controller for coupling data between a data storage system and a host includes a first processor and a first RAM coupled to the first processor; a first auxiliary processor including a memory controller and a first cache coupled to the first memory controller, the first memory controller including first interface for coupling with second auxiliary processor including second memory controller and associated second cache and second interface for coupling with first auxiliary processor, first memory controller including logic for treating the caches as single memory; a bus coupling first primary processor and first auxiliary processor; and interconnection channel separate from the bus coupling first interface of first memory controller and second interface of second memory controller. Interconnection may be an out-of-band channel permitting device-to-device sharing of associated cache memories without requiring data transfer over the bus. Method and computer program product are also provided.

Claims

1. A controller device for coupling data between a data storage system and a host computer, said controller device comprising:

a first primary processor and a first random access memory coupled to said first processor;

a first auxiliary processor including a memory controller and a first memory coupled to said first memory controller, said first memory controller including a first interface for coupling with a second auxiliary processor including a second memory controller and an associated second memory and a second interface for coupling with said first auxiliary processor, said first memory controller including control logic for treating said first memory and a second memory associated with and coupled to said second memory controller as a single memory complex;

at least one communications bus coupling said first primary processor and said first auxiliary processor; and

an interconnection channel separate from said communications bus extending between and coupling said first interface of said first memory controller and said second interface of said second memory controller.

2. The controller device in claim 1, wherein said first auxiliary processor further comprises an exclusive-OR engine.

3. The controller device in claim 2, wherein said exclusive-OR engine executes instructions performing RAID Level 5 parity calculations.

4. The controller device in claim 1, wherein said at least one communications bus comprises at least one PCI bus.

5. The controller device in claim 1, wherein said at least one communications bus comprises first and second PCI buses, said primary processor and said auxiliary processor each being coupled to each of said first and said second PCI busses.

6. The controller device in claim 5, wherein said interconnection channel comprises an out-of-band interconnect physical channel and protocol that permit device-to-device sharing their associated first and second memories without requiring data transfer over the first or second PCI busses.

7. The controller device in claim 6, wherein said interconnection physical channel is selected from the set of physical channels consisting of a serial channel, an SCI channel, a fire wire channel, a SCSI channel, an Ethernet channel, a Fibre channel, Infiniband Connection, and combinations thereof.

8. The controller device in claim 2, wherein said XOR engine comprises a programmable script-based engine allowing multiple different memory locations to be XOR'"'"'d together and the results of the XOR operation to be stored in separate memories.

9. The controller device in claim 5, wherein said first memory controller allows data to be brought in from either of the two PCI buses and stored in the first cache memory associated with said first memory controller.

10. The controller device in claim 1, wherein said first auxiliary controller is operative with a device controller has primary responsibility for a write operation and said second auxiliary controller is operative in a second controller device that is a partner controller device to said first controller device, and in connection with a write back I/O operation for said first device controller, host write data is mirrored to said partner controller over said interconnection.

11. The controller device in claim 1, wherein said primary processor, and said first and second auxiliary processors are disposed on a common substrate.

12. The controller device in claim 11, wherein said common substrate comprises a PC board.

13. The controller device in claim 1, wherein said primary processor and said first auxiliary processor are disposed on a first common substrate, and said second auxiliary processor is an external auxiliary processor disposed on a second substrate different from said first substrate.

14. The controller device in claim 1, wherein said first and said second memory operate as first and second cache memories.

16. The controller device in claim 15, wherein said XOR engine comprises a programmable script-based engine allowing multiple different memory locations to be XOR'"'"'d together and the results of the XOR operation to be stored in separate memories.

17. The controller device in claim 16, wherein said first memory controller allows data to be brought in over said PCI bus and stored in the first cache memory associated with said first memory controller.

18. The controller device in claim 15, wherein said first auxiliary controller is operative with a device controller has primary responsibility for a write operation and said second auxiliary controller is operative in a second controller device that is a partner controller device to said first controller device, and in connection with a write back I/O operation for said first device controller, host write data is mirrored to said partner controller over said interconnection.

20. The method in claim 19, wherein said single memory is identified from amongst said memories already storing some XOR operation data.

15. A controller device for coupling data between a data storage system and a host computer, said controller device comprising:

a first primary processor and a first random access memory coupled to said first processor;

a first auxiliary processor including a memory controller and an XOR processing engine executing instructions for RAID parity calculations, and a first cache memory coupled to said first memory controller, said first memory controller including a first interface for coupling with a second auxiliary processor including a second memory controller and an associated second cache memory and a second interface for coupling with said first auxiliary processor, said first memory controller including control logic for treating said first cache memory and a second cache memory associated with and coupled to said second memory controller as a single memory complex;

at least one communications PCI bus coupling said first primary processor and said first auxiliary processor; and

an interconnection channel separate from said communications bus extending between and coupling said first interface of said first memory controller and said second interface of said second memory controller, said interconnection channel comprising an out-of-band interconnect physical channel and protocol that permit device-to-device sharing of their associated first and second cache memories without requiring data transfer over said PCI bus.

19. In a computer system having a plurality of host computers, and a data storage system configured at least in part as a RAID and shared by said plurality of host computers and controlled by a plurality of controllers, each controller including at least one processor and a memory associated with and coupled to said processor, said processors being coupled by a system bus and by an interconnect different from said system bus, a method of performing a write operation to said storage system comprising:

receiving a host write command including a host write data;

identifying the memory or memories wherein are stored XOR operation data required to perform a RAID XOR operation, including memories storing relevant prior write data for a RAID stripe;

identifying a single memory from amongst all available memories to gather all of said XOR operation data;

transferring all of said XOR operation data from memories other than said identified single memory to gather all of said XOR operation data in said identified memory over said direct interconnection different from said system bus;

performing the XOR operation using said gathered XOR operation data; and

directing the write data and the parity data from said XOR operation to the RAID storage system.

21. In data storage system storing user data and parity data associated with said user data and controlled by a plurality of controllers, each of said controllers including at least one processor and an associated memory, said processors being coupled by a bus and by an interconnect different from said bus, a method of performing a write operation to said storage system comprising:

receiving a write command;

identifying memories wherein are stored parity operation data required to perform a parity operation in connection with performing said write command, said parity operation data including any relevant prior write data;

identifying a memory from amongst said identified memories to gather all of said parity operation data;

transferring all of said parity operation data to said identified memory over a direct interconnection different from a system bus;

performing the parity operation using a single processor and its associated memory; and

directing the write data and the parity data from said parity operation to the storage system.

22. The method in claim 21, wherein said write command includes write data.

23. The method in claim 21, wherein said parity operation data further includes any relevant prior write data.

26. The computer program product in claim 25, wherein said write command includes write data.

27. The method in claim 25, wherein said parity operation data further includes any relevant prior write data.

30. The apparatus in claim 29, wherein said devices comprise at least two storage device controllers.

31. The apparatus in claim 30, wherein said message comprises controller configuration information

33. The apparatus in claim 30, wherein said message comprises cache information.

34. The apparatus in claim 30, wherein said message comprises information facilitating maintaining coherency between caches within said controllers.

35. The apparatus in claim 29, wherein said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.

36. The apparatus in claim 35, wherein said computer program executes within at least one of said first and second processors.

37. The apparatus in claim 35, wherein said computer program executes within at least one of said first and second memory controllers.

38. The apparatus in claim 35, wherein said communication path selector comprises a path selector procedure implemented as first and second computer programs having instructions executing within at least one of said first and second processors and within at least one of said first and second memory controllers.

39. The apparatus in claim 29, wherein:

said devices comprise at least two storage device controllers;

said message comprises information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, and combinations thereof; and

said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.

40. The apparatus in claim 39, wherein said computer program executes within at least one of said first and second processors.

41. The apparatus in claim 39, wherein said computer program executes within at least one of said first and second memory controllers.

42. The apparatus in claim 39, wherein said communication path selector comprises a path selector procedure implemented as first and second computer programs having instructions executing within at least one of said first and second processors and within at least one of said first and second memory controllers.

24. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism, comprising:

a program module that directs a computer system having a plurality of hosts, and a data storage system configured as a RAID shared by said plurality of hosts and controlled by a plurality of controllers, to function in a specified manner, each controller including at least one processor and a memory associated with said processor, the program module including instructions for;

receiving a host write command including a host write data;

identifying the memory or memories wherein are stored all XOR operation data required to perform a RAID XOR operation, including any relevant prior write data for a RAID stripe;

identifying a single memory from amongst said identified memory or memories to gather all of said XOR operation data;

transferring all of said data from memories other than said identified single memory to said identified memory over a direct interconnection different from a system bus;

performing the XOR operation; and

directing the write data and the parity data from said XOR operation to the RAID storage.

25. A computer program product for use in conjunction with a data storage system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism, comprising:

a program module that directs a data storage system storing user data and parity data associated with said user data and controlled by a plurality of controllers each including at least one processor and an associated memory, and being coupled by a bus and by an interconnect different from said bus, to function in a specified manner, the program module including instructions for;

receiving a write command;

identifying memories wherein are stored parity operation data required to perform a parity operation in connection with performing said write command, said parity operation data including any relevant prior write data;

identifying a memory from amongst said identified memories to gather all of said parity operation data;

transferring all of said parity operation data to said identified memory over a direct interconnection different from a system bus;

performing the parity operation using a single processor and its associated memory; and

directing the write data and the parity data from said parity operation to the storage system.

29. An apparatus for communicating a message between first and second devices, each said first and second device having respective first and second processors coupled for in-band communication by a communication bus, said apparatus comprising:

a first memory controller separate from said first processor and a first memory coupled to and controlled by said first memory controller;

a second memory controller separate from said second processor and a second memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said communication bus, coupling said first and said second memory controllers for out-of-band communication between said first and second devices; and

a communication path selector directing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules.

43. An apparatus for communicating a message between first and second RAID storage device controllers, each said first and second storage device controllers having respective first and second processors coupled for in-band communication by a PCI-based communication bus, said apparatus comprising:

a first memory controller and first RAID parity operation processor separate from said first processor and a first cache memory coupled to and controlled by said first memory controller;

a second memory controller and second RAID parity operation processor separate from said second processor and a second cache memory coupled to and controlled by said second memory controller;

a communications interconnect, separate from said PCI-based communication bus, coupling said first and said second memory controllers and parity operation processors for out-of-band communication of said message between said first and second devices

said message comprises information selected from the set consisting of controller configuration information, information facilitating maintaining coherency between caches within said controllers, and combinations thereof; and

a communication path selector routing communication between said first and second devices to occur over either said communications bus or said communications interconnect according to predetermined selection rules, said selection rules including a load-leveling rule taking into account bus loading and interconnect loading and attempting to achieve a desired loading ratio.

44. The apparatus in claim 43, wherein said message comprises controller configuration information

45. The apparatus in claim 43, wherein said message comprises cache information.

46. The apparatus in claim 43, wherein said message comprises information facilitating maintaining coherency between caches within said controllers.

47. The apparatus in claim 43, wherein said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.

48. The apparatus in claim 43, wherein said desired loading ration is about a 50%-50% loading ratio.

49. The apparatus in claim 43, wherein said communication path selector comprises a path selector procedure implemented as a computer program having instructions executing within at least one of said first and second devices.