Safe message transfers on PCI-Express link from RAID controller to receiver-programmable window of partner RAID controller CPU memory

• US 7,543,096 B2
• Filed: 01/05/2006
• Issued: 06/02/2009
• Est. Priority Date: 01/20/2005
• Status: Expired due to Fees

• Current Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Original Assignee: Dot Hill Systems Corporation (Seagate Technology Holdings Plc)
• Inventors: Davies, Ian Robert
• Primary Examiner(s): Rinehart; Mark
• Assistant Examiner(s): Spittle; Matthew D
• Application Number:
  US11/329,470
• Publication Number:
  US 20060161709A1
• Time in Patent Office: 1,244 Days
• Field of Search: 709/216, 710/268, 714 5- 8
• US Class Current: 710/268
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 13/404 with address mapping

Abstract

A fault-tolerant mass storage system includes two RAID controllers that communicate via a PCI-Express link. Each controller has a bus bridge coupled to the link, a cache memory that caches user data for storage on disk drives controlled by the controllers, and a CPU. The CPU fetches and executes program instructions from a CPU memory coupled to it. The CPU programs the bus bridge with window information defining a window of locations within the CPU memory, which is less than an entirety of the CPU memory. The bus bridge receives data on the link from the other controller and if the header of a packet containing the data indicates it is destined for the CPU memory, the bus bridge translates the address of the data so as to write the data safely to the CPU memory, but only within the window and nowhere else within the CPU memory.

Claims

1. A fault-tolerant mass storage system, comprising:

first and second RAID controllers, coupled for communication via a PCI-Express link, each comprising;

a bus bridge, coupled to said link and to first and second buses;

a cache memory, coupled to said first bus, configured to cache user data for storage on disk drives controlled by said controllers; and

a CPU, and a CPU memory coupled to said CPU, each coupled to said second bus, wherein said CPU is configured to fetch and execute program instructions from said CPU memory, wherein said CPU is configured to program said bus bridge with window information defining a window of locations within said CPU memory, wherein said window comprises less than an entirety of said CPU memory;

wherein said bus bridge is configured to receive data on said link from the other of said first and second RAID controllers, to write said data to said CPU memory if destined for said CPU memory, but only within said window and nowhere else within said CPU memory, and to write said data to said cache memory if destined for said cache memory, wherein said bus bridge is configured to refrain from writing said data to said CPU memory outside of said window even if said bus bridge determines that a portion of said data is destined for said CPU memory within said window.

2. The system of claim 1, wherein said bus bridge further comprises:

at least one control register, programmable by said CPU with said window information specifying said window of locations within said CPU memory.

3. The system of claim 2, wherein said bus bridge is configured to reset to a predetermined reset value of said window information in said control register until programmed by said CPU, wherein if said bus bridge receives said data prior to said CPU programming a value other than said reset value into said control register, said bus bridge is configured to refrain from writing said data to said CPU memory even if said bus bridge determines that said data is destined for said CPU memory.

4. The system of claim 1, wherein said bus bridge is configured to receive a PCI-Express memory write request transaction layer packet (TLP) on said link from the other RAID controller, said TLP including a payload field containing said data, wherein said bus bridge is configured to determine from a header of said TLP whether said data is destined for said CPU memory, and to determine from said header whether said data is destined for said cache memory.

5. The system of claim 4, wherein said header includes an address field for storing a first address, wherein if said data is destined for said CPU memory, said bus bridge is configured to translate said first address using said window information into a second address that is within said window, and to write said data to said CPU memory at said second address only within said window.

6. The system of claim 5, wherein if said data is destined for said cache memory, said bus bridge is configured to write said data to said cache memory at said first address stored in said address field.

7. The system of claim 6, wherein said cache memory occupies a predetermined address range within a memory address space defined by said address field, wherein said bus bridge is configured to determine whether said data is destined for said cache memory by determining whether said first address is within said cache memory address range.

8. The system of claim 5, wherein said CPU memory occupies a predetermined address range within a memory address space defined by said address field, wherein said bus bridge is configured to determine whether said data is destined for said CPU memory by determining whether said first address is within said CPU memory address range.

9. The system of claim 5, wherein said window information comprises upper address bits programmed into said bus bridge by said CPU, wherein said bus bridge is configured to translate said first address into said second address by replacing a portion of upper address bits of said first address with said upper address bits of said window information.

10. The system of claim 5, wherein said window information comprises a base address of said window, wherein said bus bridge is configured to translate said first address into said second address by adding said base address to said first address.

11. The system of claim 10, wherein said bus bridge further comprises:

a subtrahend, programmable by said CPU;

wherein said bus bridge on the other of the first and second RAID controllers transmitting said TLP on said link is configured to subtract said subtrahend from a source address of said data in said CPU memory of said transmitting RAID controller to generate said first address, wherein said first address comprises an offset into said window.

12. The system of claim 10, wherein said bus bridge is configured to determine whether said data is destined for said CPU memory by determining whether a predetermined bit of the TLP header has a predetermined value.

13. The system of claim 12, wherein the PCI-Express memory write request TLP has a 4 double word header with data format, wherein said predetermined bit of the TLP header is one of bits 63 through 32 of the address field.

14. The system of claim 5, wherein said header includes a length field for storing a length of said data, wherein if said length specifies one or more locations outside said window, said bus bridge is configured to refrain from writing said data to said CPU memory outside said window.

15. The system of claim 1, wherein said window information specifies a size of said window.

16. The system of claim 1, wherein said window information comprises a base address of said window within said CPU memory.

17. The system of claim 1, wherein said first and second RAID controllers operate as an active-active redundant pair.

18. The system of claim 1, wherein said data destined for said CPU memory comprises a message from said first RAID controller to said second RAID controller for managing configuration of disk arrays coupled to and controlled by said first and second RAID controllers.

19. The system of claim 1, wherein said data destined for said CPU memory comprises a message from said first RAID controller to said second RAID controller for managing input/output operations to disk arrays coupled to and controlled by said first and second RAID controllers.

20. The system of claim 1, wherein said data destined for said CPU memory comprises a message from said first RAID controller to said second RAID controller for managing configuration of said first and second RAID controllers.

21. The system of claim 1, wherein said data destined for said CPU memory comprises a message from said first RAID controller to said second RAID controller for managing said cache memory of said second RAID controller.

22. The system of claim 1, wherein said bus bridge is further coupled to a third bus, for coupling to a disk interface, wherein said disk interface is configured to control disk arrays controlled by said first and second RAID controllers.

23. The system of claim 22, wherein said bus bridge is further coupled to a fourth bus, for coupling to a host interface, wherein said host interface is configured to communicate with host computers coupled to said first and second RAID controllers.

24. The system of claim 1, wherein each of said first and second RAID controllers further comprises:

a second bus bridge, coupled to said second bus, and to said CPU and said CPU memory, wherein said CPU fetches and executes said program instructions from said CPU memory via said second bus bridge, wherein said bus bridge writes said data to said window within said CPU memory via said second bus bridge.

25. The system of claim 1, wherein said bus bridge comprises a direct memory access controller (DMAC), wherein said CPU on the other of the first and second RAID controllers transmitting said TLP on said link is configured to program said DMAC to transfer said data from said CPU memory of said transmitting RAID controller on said link to said bus bridge of said receiving RAID controller for writing into said CPU memory within said window.

26. A method for reliably transferring data between first and second RAID controllers via a PCI-Express link in a fault-tolerant mass storage system, the first RAID controller having a first bus bridge coupled to the link, to a first cache memory for caching user data for storage on disk drives controlled by the first RAID controller, to a first CPU, and to a first CPU memory for storing program instructions fetched and executed by the first CPU, the second RAID controller having a second bus bridge coupled to the link, to a second cache memory for caching user data for storage on disk drives controlled by the second RAID controller, to a second CPU, and to a second CPU memory for storing program instructions fetched and executed by the second CPU, the method comprising:

programming, by the first CPU, the first bus bridge with window information defining a window of locations within the first CPU memory, wherein the window comprises less than an entirety of the first CPU memory;

receiving, by the first bus bridge, data on the link from the second bus bridge;

writing, by the first bus bridge, the data to the first CPU memory if the data is destined for the first CPU memory, but only within the window and nowhere else within the first CPU memory;

refraining, by the first bus bridge, from writing the data to the first CPU memory outside of the window even if the first bus bridge determines that a portion of the data is destined for the first CPU memory within the window; and

writing, by the first bus bridge, the data to the first cache memory if the data is destined for the first cache memory.

27. The method of claim 26, further comprising:

programming, by the first CPU, at least one control register of the first bus bridge with the window information specifying the window of locations within the first CPU memory.

28. The method of claim 27, further comprising:

resetting, by the first bus bridge, to a predetermined reset value of the window information in the control register until programmed by the first CPU;

determining, by the first bus bridge, whether the data has been received prior to the first CPU programming a value other than the reset value into the control register; and

refraining, by the first bus bridge, from writing the data to the first CPU memory, if the data has been received prior to the first CPU programming a value other than the reset value into the control register, even if the first bus bridge determines that the data is destined for the first CPU memory.

29. The method of claim 26, further comprising:

receiving, by the first bus bridge, a PCI-Express memory write request transaction layer packet (TLP) on the link from the second bus bridge, the TLP including a payload field containing the data;

determining, by the first bus bridge, from a header of the TLP whether the data is destined for the first CPU memory; and

determining, by the first bus bridge, from the header whether the data is destined for the first cache memory.

30. The method of claim 29, wherein the header includes an address field for storing a first address, the method further comprising:

if the data is destined for the first CPU memory;

translating, by the first bus bridge, the first address using the window information into a second address that is within the window; and

writing, by the first bus bridge, the data to the first CPU memory at the second address only within the window.

31. The method of claim 30, further comprising:

writing, by the first bus bridge, the data to the first cache memory at the first address stored in the address field, if the data is destined for the first cache memory.

32. The method of claim 31, wherein the first cache memory occupies a predetermined address range within a memory address space defined by the address field, wherein the first bus bridge determining whether the data is destined for the first cache memory comprises determining whether the first address is within the first cache memory address range.

33. The method of claim 30, wherein the first CPU memory occupies a predetermined address range within a memory address space defined by the address field, wherein the first bus bridge determining whether the data is destined for the first CPU memory comprises determining whether the first address is within the first CPU memory address range.

34. The method of claim 30, wherein the window information comprises upper address bits programmed into the first bus bridge by the first CPU, wherein the first bus bridge translating the first address into the second address comprises replacing a portion of upper address bits of the first address with the upper address bits of the window information.

35. The method of claim 30, wherein the window information comprises a base address of the window, wherein the first bus bridge translating the first address into the second address comprises adding the base address to the first address.

36. The method of claim 35, further comprising:

programming, by the second CPU, a subtrahend into the second bus bridge;

subtracting, by the second bus bridge, the subtrahend from a source address of the data in the second CPU memory to generate the first address, wherein the first address comprises an offset into the window.

37. The method of claim 35, wherein the first bus bridge determining whether the data is destined for the first CPU memory comprises determining whether a predetermined bit of the TLP header has a predetermined value.

38. The method of claim 30, wherein the header includes a length field for storing a length of the data, the method further comprising:

refraining, by the first bus bridge, from writing the data to the first CPU memory outside the window, if the length specifies one or more locations outside the window.

39. The method of claim 26, wherein the first and second bus bridges comprise a direct memory access controller (DMAC), the method further comprising:

programming, by the second CPU, the second DMAC to transfer the data from the second CPU memory of the second RAID controller on the link to the first bus bridge for writing into the first CPU memory within the window.

40. A bus bridge, for instantiation on each of two redundant array of inexpensive disks (RAID) controllers coupled for communication on a PCI-Express link, the bus bridge comprising:

a PCI-Express interface, for coupling to said link, wherein said PCI-Express interface is configured to receive data on said link from said PCI-Express interface of the other RAID controller;

a memory bus interface, for coupling to a cache memory, configured to cache user data for storage on disk drives controlled by said controllers;

a local bus interface, for coupling to a CPU and to a CPU memory from which said CPU fetches and executes program instructions;

at least one control register, programmable by said CPU with window information defining a window of locations within said CPU memory, said window comprising less than an entirety of said CPU memory; and

control logic, coupled to said interfaces, wherein said control logic is configured to determine whether said received data is destined for said CPU memory, and if so, to control said local bus interface to write said data to said CPU memory, but only within said window and nowhere else within said CPU memory, and to determine whether said data is destined for said cache memory, and if so, to control the memory bus interface to write said data to said cache memory, wherein said control logic is configured to refrain from controlling said memory bus interface to write said data to said CPU memory outside of said window even if said control logic determines that a portion of said data is destined for said CPU memory within said window.

41. The bus bridge of claim 40, wherein said bus bridge further comprises:

at least one control register, programmable by said CPU with said window information specifying said window of locations within said CPU memory.

42. The bus bridge of claim 41, further configured to reset to a predetermined reset value of said window information in said control register until programmed by said CPU, wherein if said bus bridge receives said data prior to said CPU programming a value other than said reset value into said control register, said control logic is configured to refrain from writing said data to said CPU memory even if said control logic determines that said data is destined for said CPU memory.

43. The bus bridge of claim 40, wherein said PCI-Express interface is configured to receive a PCI-Express memory write request transaction layer packet (TLP) on said link from the other RAID controller, said TLP including a payload field containing said data, wherein said control logic is configured to determine from a header of said TLP whether said data is destined for said CPU memory, and to determine from said header whether said data is destined for said cache memory.

44. The bus bridge of claim 43, wherein said header includes an address field for storing a first address, wherein if said control logic determines said data is destined for said CPU memory, said control logic translates said first address using said window information into a second address that is within said window, and controls said memory bus interface to write said data to said CPU memory at said second address only within said window.

45. The bus bridge of claim 44, wherein if said control logic determines said data is destined for said cache memory, said control logic controls said memory bus interface to write said data to said cache memory at said first address stored in said address field.

46. The bus bridge of claim 45, wherein said cache memory occupies a predetermined address range within a memory address space defined by said address field, wherein said control logic is configured to determine whether said data is destined for said cache memory by determining whether said first address is within said cache memory address range.

47. The bus bridge of claim 44, wherein said CPU memory occupies a predetermined address range within a memory address space defined by said address field, wherein said control logic is configured to determine whether said data is destined for said CPU memory by determining whether said first address is within said CPU memory address range.

48. The bus bridge of claim 44, wherein said window information comprises upper address bits programmed into said bus bridge by said CPU, wherein said control logic is configured to translate said first address into said second address by replacing a portion of upper address bits of said first address with said upper address bits of said window information.

49. The bus bridge of claim 44, wherein said window information comprises a base address of said window, wherein said control logic is configured to translate said first address into said second address by adding said base address to said first address.

50. The bus bridge of claim 49, wherein said bus bridge further comprises:

a subtrahend, programmable by said CPU;

wherein said control logic of the RAID controller transmitting the TLP on the link is configured to subtract said subtrahend from a source address of said data in said CPU memory of the other RAID controller to generate said first address, wherein said first address comprises an offset into said window.

51. The bus bridge of claim 49, wherein said control logic is configured to determine whether said data is destined for said CPU memory by determining whether a predetermined bit of the TLP header has a predetermined value.

52. The bus bridge of claim 44, wherein said header includes a length field for storing a length of said data, wherein if said control logic determines said length specifies one or more locations outside said window, said control logic is configured to refrain from controlling said memory bus interface to write said data to said CPU memory outside said window.

53. The bus bridge of claim 40, wherein said window information specifies a size of said window.

54. The bus bridge of claim 40, wherein said window information comprises a base address of said window within said CPU memory.

55. The bus bridge of claim 40, further comprising:

a second local bus interface, for coupling to a disk interface, wherein said disk interface is configured to control disk arrays controlled by said RAID controllers.

56. The bus bridge of claim 55, further comprising:

a third local bus interface, for coupling to a host interface, wherein said host interface is configured to communicate with host computers coupled to said RAID controllers.

57. The bus bridge of claim 40, further comprising:

a direct memory access controller (DMAC), configured for programming by said CPU to transfer said data from said CPU memory on said link to said bus bridge.

58. The bus bridge of claim 40, further comprising:

a first-in-first-out (FIFO) memory, coupling said memory interface and said local bus interface, for buffering said data.

59. The bus bridge of claim 40, wherein said local bus interface comprises a PCI bus interface.

60. The bus bridge of claim 40, wherein said local bus interface comprises a PCI-X bus interface.

61. A fault-tolerant mass storage system, comprising:

first and second RAID controllers, coupled for communication via a PCI-Express link, each comprising;

a bus bridge, coupled to said link;

a CPU memory, coupled to said bus bridge, configured to store program instructions and to cache user data for storage on disk drives controlled by said controllers; and

a CPU, coupled to said CPU memory and to said bus bridge, configured to fetch and execute said program instructions from said CPU memory, to control caching of said user data in said CPU memory, and to program said bus bridge with window information defining a window of locations within said CPU memory, wherein said window comprises less than an entirety of said CPU memory;

wherein said bus bridge is configured to receive data on said link from the other of said first and second RAID controllers, and to write said data to said CPU memory, but only within said window and nowhere else within said CPU memory, wherein said bus bridge is configured to refrain from writing said data to said CPU memory outside of said window even if said bus bridge determines that a portion of said data is destined for said CPU memory within said window.

62. A method for reliably transferring data between first and second RAID controllers via a PCI-Express link in a fault-tolerant mass storage system, each of the RAID controllers having a bus bridge coupled to the link, to a CPU, and to a CPU memory for storing program instructions fetched and executed by the CPU and for caching user data for storage on disk drives controlled by the RAID controllers, the method comprising:

programming, by the first CPU, the first bus bridge with window information defining a window of locations within the first CPU memory, wherein the window comprises less than an entirety of the CPU memory;

receiving, by the first bus bridge, data on the link from the second bus bridge;

writing, by the first bus bridge, the data to the first CPU memory, but only within the window and nowhere else within the first CPU memory; and

refraining, by the first bus bridge, from writing the data to the first CPU memory outside of the window even if the first bus bridge determines that a portion of the data is destined for the first CPU memory within the window.

63. A bus bridge, for instantiation on each of two redundant array of inexpensive disks (RAID) controllers coupled for communication on a PCI-Express link, the bus bridge comprising:

a PCI-Express interface, for coupling to said link, wherein said PCI-Express interface is configured to receive data on said link from said PCI-Express interface of the other RAID controller;

a local bus interface, for coupling to a CPU and to a CPU memory from which said CPU fetches and executes program instructions and which caches user data for storage on disk drives controlled by the controllers;

at least one control register, programmable by said CPU with window information defining a window of locations within said CPU memory, said window comprising less than an entirety of said CPU memory; and

control logic, coupled to said interfaces, wherein said control logic is configured to control said local bus interface to write said data to said CPU memory, but only within said window and nowhere else within said CPU memory, wherein said control logic is configured to refrain from controlling said local bus interface to write said data to said CPU memory outside of said window even if said control logic determines that a portion of said data is destined for said CPU memory within said window.

64. A bus bridge, for instantiation on each of two redundant array of inexpensive disks (RAID) controllers coupled for communication on a PCI-Express link, the bus bridge comprising:

a PCI-Express interface, for coupling to said link, wherein said PCI-Express interface is configured to receive data on said link from said PCI-Express interface of the other RAID controller;

a first bus interface, for coupling to a CPU;

a second bus interface, for coupling to a CPU memory from which said CPU fetches and executes program instructions and which caches user data for storage on disk drives controlled by the controllers;

at least one control register, programmable by said CPU with window information defining a window of locations within said CPU memory, said window comprising less than an entirety of said CPU memory; and

control logic, coupled to said interfaces, wherein said control logic is configured to control said second bus interface to write said data to said CPU memory, but only within said window and nowhere else within said CPU memory, wherein said control logic is configured to refrain from controlling said second bus interface to write said data to said CPU memory outside of said window even if said control logic determines that a portion of said data is destined for said CPU memory within said window.