Methods and apparatus for coordinating shared multiple raid controller access to common storage devices

• US 6,073,218 A
• Filed: 12/23/1996
• Issued: 06/06/2000
• Est. Priority Date: 12/23/1996
• Status: Expired due to Term

• Current Assignee: NetApp, Inc.
• Original Assignee: LSI Logic Corporation (Broadcom, Inc.)
• Inventors: Fredin, Gerald J., DeKoning, Rodney A.
• Primary Examiner(s): Nguyen, Hiep T
• Application Number:
  US08/772,614
• Time in Patent Office: 1,261 Days
• Field of Search: 711/114, 711/152, 711/145, 711/148, 711/149, 711/150, 711/151, 711/153, 711/168, 711/162, 714/6, 714/11, 710/20, 710/21, 710/38, 710/241
• US Class Current: 711/150
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 9/52 Program synchronisation; Mutual exclusion, e.g. by means of semaphores

Abstract

Methods and associated apparatus for performing concurrent I/O operations on a common shared subset of disk drives (LUNs) by a plurality of RAID controllers. The methods of the present invention are operable in all of a plurality of RAID controllers to coordinate concurrent access to a shared set of disk drives. In addition to providing redundancy features, the plurality of RAID controllers operable in accordance with the methods of the present invention enhance the performance of a RAID subsystem by better utilizing available processing power among the plurality of RAID controllers. Under the methods of the present invention, each of a plurality of RAID controllers may actively process different I/O requests on a common shared subset of disk drives. One of the plurality of controllers is designated as primary with respect to a particular shared subset of disk drives. The plurality of RAID controllers then exchange messages over a communication medium to coordinate concurrent access to the shared subset of disk drives through the primary controller. The messages exchanged include semaphore lock and release requests to coordinate exclusive access during critical operations as well as cache and meta-cache data to maintain cache coherency between the plurality of the RAID controllers with respect to the common shared subset of disk drives. These messages are exchanged via any of several well known communication mediums including, a shared memory common to the plurality of controllers and the communication bus connecting the shared subset of disk drives to each of the plurality of controllers.

Claims

1. In a system including a plurality of RAID control modules connected to a common shared LUN in a RAID storage subsystem, a method operable within said plurality of RAID control modules for coordinating exclusive access by said plurality of control modules to said common shared LUN comprising the steps of:

designating one of said plurality of RAID control modules as a primary control module; and

exchanging access coordination messages between said primary control module and others of said plurality of RAID control modules to coordinate temporary exclusive access by one of said plurality of RAID control modules to identified stripes within said common shared LUN.

2. The method of claim 1 wherein the step of exchanging access coordination messages includes the steps of:

generating, within one of said others of said plurality of RAID control modules, an exclusive access request message;

transmitting said exclusive access request message to said primary control module; and

awaiting receipt by said one of said others of said plurality of RAID control modules of an exclusive access grant message from said primary control module granting exclusive access to said common shared LUN.

3. The method of claim 2 wherein the step of exchanging access coordination messages further includes the steps of:

receiving, within said primary control module, said exclusive access request message;

awaiting availability of exclusive access to said common shared LUN in response to receipt of said exclusive access request message;

generating, within said primary control module, said exclusive access grant message in response to availability of exclusive access to said common shared LUN; and

transmitting said exclusive access grant message to said one of said others of said plurality of RAID control modules.

4. The method of claim 3 wherein said exclusive access request message identifies a portion of said common shared LUN for which exclusive access is desired and wherein the step of awaiting availability comprises the step of awaiting availability of exclusive access to the portion of said common shared LUN identified by the received exclusive access request message.

5. The method of claim 4 wherein said primary control module is associated with at least one semaphore and wherein the step of awaiting availability includes the steps of:

determining that none of said at least one semaphore is associated with said portion of said common shared LUN;

associating one of said at least one semaphore with said portion in response to the determination none of said at least one semaphore is associated with said portion; and

locking the semaphore associated with said portion.

6. The method of claim 3 wherein the step of exchanging access coordination messages further includes the steps of:

generating, within said one of said others of said plurality of RAID control modules, an exclusive access release message; and

transmitting said exclusive access release message to said primary control module.

7. The method of claim 6 wherein the step of exchanging access coordination messages further includes the steps of:

receiving, within said primary control module, said exclusive access release message; and

releasing exclusive access to said common shared LUN in response to receipt of said exclusive access release message.

8. The method of claim 7 wherein said exclusive access request message identifies a portion of said common shared LUN for which exclusive access is desired and wherein the step of awaiting availability comprises the step of awaiting availability of exclusive access to the portion of said common shared LUN identified by the received exclusive access request message.

9. The method of claim 8 wherein said primary control module is associated with at least one semaphore and wherein the step of releasing exclusive access includes the steps of:

determining, within said primary control module, which of said at least one semaphore was previously associated with said portion of said common shared LUN; and

unlocking the semaphore previously associated with said portion.

10. The method of claim 1 wherein each of said plurality of RAID control modules is associated with a cache memory subsystem and wherein the method further comprises the step of:

exchanging cache coherency messages among said plurality of RAID control modules to maintain coherency of data in said cache memory subsystem associated with each of said plurality of RAID control modules.

11. The method of claim 10 wherein said cache coherency messages include cache data generated in response to processing of an I/O request from an attached host computer.

12. The method of claim 10 wherein the step of exchanging cache coherency messages includes the steps of:

generating, within one of said others of said plurality of RAID control modules having a local cache memory, a cache update permission request message;

transmitting said cache update permission request message to said primary control module; and

awaiting receipt by said one of said others of said plurality of RAID control modules of a cache update permission grant message from said primary control module granting permission to update said local cache memory.

13. The method of claim 12 wherein the step of exchanging cache coherency messages further includes the steps of:

receiving, within said primary control module, said cache update permission request message;

invalidating local cache memory in said others of said plurality of RAID control modules in response to receiving said cache update permission request message;

generating, within said primary control module, said cache update permission grant message in response to invalidating local cache memory; and

transmitting said exclusive access grant message to said one of said others of said plurality of RAID control modules.

14. The method of claim 13 wherein said cache update permission request message identifies a portion of said common shared LUN for which update of said local cache memory update is requested and wherein the step of invalidating comprises the step of:

for each of said others of said plurality of RAID control modules performing the steps of;

determining whether said each of said others of said plurality of RAID control modules have portions of its respective local cache memory corresponding to said portion of said common shared LUN;

generating a cache invalidate message identifying said portion of said respective local cache memory in response to the determination that said each of said others of said plurality of RAID control modules has a portion of its respective local cache memory corresponding to said portion of said common shared LUN; and

transmitting said cache invalidate message to said each of said others of said plurality of RAID control modules in response to generation of said cache invalidate message.

15. The method of claim 14 wherein the step of invalidating further comprises the steps of:

receiving, within said each of said others of said RAID control modules, said cache invalidate message; and

invalidating the identified portion of said respective local cache memory within said each of said others of said RAID control modules.

16. In a system including a plurality of storage control modules connected to a common shared set of storage devices in a storage subsystem, a method operable within said plurality of storage control modules for processing I/O requests directed to said common shared set of storage devices comprising the steps of:

designating one of said plurality of storage control modules as a primary control modules;

receiving a plurality of I/O requests from attached host computer systems;

coordinating temporary exclusive access to identified stripes within said common shared set of storage devices by said distinct control modules, wherein the step of coordinating includes the step of;

exchanging exclusive access messages between said primary control module and others of said plurality of storage control modules; and

processing each of said plurality of I/O requests substantially in parallel within distinct control modules of said plurality of storage control modules.

17. The method of claim 16:

wherein said exclusive access messages include exclusive access request messages directed from said distinct control modules to said primary control module,

wherein said exclusive access messages include exclusive access grant messages directed from said primary control module to said distinct control modules, and

wherein said exclusive access messages include exclusive access release messages directed from said distinct control modules to said primary control module.

18. The method of claim 16 wherein each of said plurality of storage control modules is associated with a cache memory subsystem and wherein the step of coordinating includes the step of:

exchanging cache information among said plurality of storage control modules to maintain coherency of information stored in said cache memory associated with each of said plurality of storage control modules.

19. A program storage device readable by a computer, tangibly embodying a program or instructions executable by computers in a plurality of RAID control modules connected to a common shared LUN in a RAID storage subsystem to perform the method steps for coordinating exclusive access by said plurality of control modules to said common shared LUN, said method steps comprising:

designating one of said plurality of RAID control modules as a primary control module; and

exchanging access coordination messages between said primary control module and others of said plurality of RAID control modules to coordinate temporary exclusive access by one of said plurality of RAID control modules to identified stripes within said common shared LUN.

20. The program storage device of claim 19 wherein the method step of exchanging access coordination messages includes the steps of:

generating, within one of said others of said plurality of RAID control modules, an exclusive access request message;

transmitting said exclusive access request message to said primary control module; and

awaiting receipt by said one of said others of said plurality of RAID control modules of an exclusive access grant message from said primary control module granting exclusive access to said common shared LUN.

21. The program storage device of claim 20 wherein the method step of exchanging access coordination messages further includes the steps of:

receiving, within said primary control module, said exclusive access request message;

awaiting availability of exclusive access to said common shared LUN in response to receipt of said exclusive access request message;

generating, within said primary control module, said exclusive access grant message in response to availability of exclusive access to said common shared LUN; and

transmitting said exclusive access grant message to said one of said others of said plurality of RAID control modules.

22. The program storage device of claim 21 wherein said exclusive access request message identifies a portion of said common shared LUN for which exclusive access is desired and wherein the method step of awaiting availability comprises the step of awaiting availability of exclusive access to the portion of said common shared LUN identified by the received exclusive access request message.

23. The program storage device of claim 22 wherein said primary control module is associated with at least one semaphore and wherein the method step of awaiting availability includes the steps of:

determining that none of said at least one semaphore is associated with said portion of said common shared LUN;

associating one of said at least one semaphore with said portion in response to the determination none of said at least one semaphore is associated with said portion; and

locking the semaphore associated with said portion.

24. The program storage device of claim 21 wherein the method step of exchanging access coordination messages further includes the steps of:

generating, within said one of said others of said plurality of RAID control modules, an exclusive access release message; and

transmitting said exclusive access release message to said primary control module.

25. The program storage device of claim 24 wherein the method step of exchanging access coordination messages further includes the steps of:

receiving, within said primary control module, said exclusive access release message; and

releasing exclusive access to said common shared LUN in response to receipt of said exclusive access release message.

26. The program storage device of claim 25 wherein said exclusive access request message identifies a portion of said common shared LUN for which exclusive access is desired and wherein the method step of awaiting availability comprises the step of awaiting availability of exclusive access to the portion of said common shared LUN identified by the received exclusive access request message.

27. The program storage device of claim 26 wherein said primary control module is associated with at least one semaphore and wherein the method step of releasing exclusive access includes the steps of:

determining, within said primary control module, which of said at least one semaphore was previously associated with said portion of said common shared LUN; and

unlocking the semaphore previously associated with said portion.

28. The program storage device of claim 19 wherein each of said plurality of RAID control modules is associated with a cache memory subsystem and wherein the method further comprises the step of:

exchanging cache coherency messages among said plurality of RAID control modules to maintain coherency of data in said cache memory subsystem associated with each of said plurality of RAID control modules.

29. The program storage device of claim 28 wherein said cache coherency messages include cache data generated in response to processing of an I/O request from an attached host computer.

30. The program storage device of claim 28 wherein the method step of exchanging cache coherency messages includes the steps of:

generating, within one of said others of said plurality of RAID control modules having a local cache memory, a cache update permission request message;

transmitting said cache update permission request message to said primary control module; and

awaiting receipt by said one of said others of said plurality of RAID control modules of a cache update permission grant message from said primary control module granting permission to update said local cache memory.

31. The program storage device of claim 30 wherein the method step of exchanging cache coherency messages further includes the steps of:

receiving, within said primary control module, said cache update permission request message;

invalidating local cache memory in said others of said plurality of RAID control modules in response to receiving said cache update permission request message;

generating, within said primary control module, said cache update permission grant message in response to invalidating local cache memory; and

transmitting said exclusive access grant message to said one of said others of said plurality of RAID control modules.

32. The program storage device of claim 31 wherein said cache update permission request message identifies a portion of said common shared LUN for which update of said local cache memory update is requested and wherein the method step of invalidating comprises the step of:

for each of said others of said plurality of RAID control modules performing the steps of;

determining whether said each of said others of said plurality of RAID control modules have portions of its respective local cache memory corresponding to said portion of said common shared LUN;

generating a cache invalidate message identifying said portion of said respective local cache memory in response to the determination that said each of said others of said plurality of RAID control modules has a portion of its respective local cache memory corresponding to said portion of said common shared LUN; and

transmitting said cache invalidate message to said each of said others of said plurality of RAID control modules in response to generation of said cache invalidate message.

33. The program storage device of claim 32 wherein the method step of invalidating further comprises the steps of:

receiving, within said each of said others of said RAID control modules, said cache invalidate message; and

invalidating the identified portion of said respective local cache memory within said each of said others of said RAID control modules.

34. A program storage device readable by a computer, tangibly embodying a program or instructions executable by computers in a plurality of storage control modules connected to a common shared set of storage devices in a storage subsystem to perform the method steps for processing I/O requests directed to said common shared set of storage devices, said method steps comprising:

designating one of said plurality of storage control modules as a primary control modules;

receiving a plurality of I/O requests from attached host computer systems;

coordinating temporary exclusive access to identified stripes within said common shared set of storage devices by said distinct control modules, wherein the step of coordinating includes the step of;

exchanging exclusive access messages between said primary control module and others of said plurality of storage control modules; and

processing each of said plurality of I/O requests substantially in parallel within distinct control modules of said plurality of storage control modules.

35. The program storage device of claim 34:

wherein said exclusive access messages include exclusive access request messages directed from said distinct control modules to said primary control module,

wherein said exclusive access messages include exclusive access grant messages directed from said primary control module to said distinct control modules, and

wherein said exclusive access messages include exclusive access release messages directed from said distinct control modules to said primary control module.

36. The program storage device of claim 34 wherein each of said plurality of storage control modules is associated with a cache memory subsystem and wherein the method step of coordinating includes the step of:

exchanging cache information among said plurality of storage control modules to maintain coherency of information stored in said cache memory associated with each of said plurality of storage control modules.