System and method for maintaining cache coherency and data synchronization in a computer system having multiple active controllers

• US 6,247,099 B1
• Filed: 06/03/1999
• Issued: 06/12/2001
• Est. Priority Date: 06/03/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Skazinski, Joseph, McKean, Brian, Otterness, Noel S.
• Primary Examiner(s): NGUYEN, THAN VINH
• Application Number:
  US09/325,033
• Time in Patent Office: 740 Days
• Field of Search: 711/117, 711/118, 711/141, 711/144, 711/145
• US Class Current: 711/141
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache

Abstract

System, method and computer program for maintaining cache coherency amongst a plurality of caching storage controllers operating in unison supplying data in response to requests from one or more host computers. The method comprises the steps of defining a reservation data structure to maintain reserved, partial, and full ownership status of data extents that are part of the logical unit or storage volume, and using the reservation data structure to verify that a new update to the data is allowed.

Claims

1. A method of maintaining cache coherency amongst a plurality of caching storage controllers operating in unison in a dual-active configuration and supplying data in response to requests from one or more host computers;

said method comprising the steps;

defining a storage volume reservation table data structure to maintain reserved, partial, and full ownership status of data extents that are part of the logical unit or storage volume, said ownership status applying to all or a portion of a logical unit or storage volume;

said storage volume reservation table having a plurality of reservation table entries indexed by storage volume number, each said reservation table entry including an ownership field, a hash table of locks, and a linked list of currently blocked data extents; and

using said storage volume reservation table data structure to verify that a new update to the data is allowed.

2. The method in claim 1, wherein said ownership field specifying either that the controller in which the SVRT resides holds the reservation for this storage volume, or that another controller reserves the storage volume.

3. The method in claim 2, wherein said ownership field including an additional qualifier that identifies a data sector lock status as one of:

(a) the storage volume contains a full lock of all data sectors, (b) the storage volume contains one or more number partial locks, or (c) certain data sectors are temporarily blocked.

4. The method in claim 1, wherein said hash table of locks field being indexed by hash entry value.

5. The method in claim 4, wherein said hash entry value being a value between and the total number of entries in the hash table of locks.

6. The method in claim 5, wherein as a lock is established it is assigned to a hash entry value calculated using a modulo operation on the start sector value of the lock.

7. The method in claim 1, wherein each said storage volume reservation table including a storage volume identifier, a plurality of storage volume state parameters including parameters indicating that the lock is full, partial, or a block and that the reservation is local or remote, and a hash table of locks.

8. The method in claim 7, wherein said storage volume state field within said storage volume reservation entry data structure providing a set of six flags to allow a quick check to be made to determine the state of reservations for a storage volume;

said flags indicating that there are no locks, partial locks, block lock, local reservation, or remote reservation.

9. The method in claim 8, wherein said no lock status indicating that nobody currently owns any locks;

said partial lock status indicating that the controller in which the storage volume reservation table is stored owns pieces of the storage volume, but not the entire storage volume;

said block lock status indicating a temporary blocking lock is in place for this storage volume and usually indicating that another controller is performing an operation;

said local reservation status indicating that this controller has a reserve command outstanding for this storage volume, and said remote reservation status indicating that another controller has a reserve command outstanding for this storage volume.

10. The method in claim 8, wherein each caching storage controller storing a my block locks list containing all of the outstanding blocking locks which have been granted to this controller.

11. The method in claim 1, wherein each caching storage controller storing a remote locks pending list containing all of the outstanding lock requests for this storage volume made by other controllers.

12. The method in claim 1, wherein each caching storage controller storing a local locks pending list containing all of the outstanding lock requests made by this controller to other controllers.

13. The method in claim 1, wherein each caching storage controller storing a data extent hash table containing all of the granted partial locks for this controller.

14. The method in claim 1, wherein each caching storage controller storing a block locks list containing all of the outstanding blocking locks which have been granted to other controllers;

a block lock flag being set in the storage volume state field in there are any entries in the block locks list.

15. The method in claim 1, wherein each caching storage controller storing:

a remote locks pending list containing all of the outstanding lock requests for this storage volume made by other controllers;

a local locks pending list containing all of the outstanding lock requests made by this controller to other controllers;

a data extent hash table containing all of the granted partial locks for this controller;

a block locks list containing all of the outstanding blocking locks which have been granted to other controllers, a block lock flag being set in the storage volume state field in there are any entries in the block locks list; and

a my block locks list containing all of the outstanding blocking locks which have been granted to this controller.

16. The method in claim 1, further comprising the step of defining a storage volume ownership transaction data structure established in a memory of each controller and tracking the state of a current transaction between the controller in which the data structure is established and an alternate controller.

17. The method in claim 16, wherein said storage volume ownership transaction data structure storing a transaction value, a current status, a time stamp value, a reference to a host command, and a reference to the program code that handles transaction completion.

18. The method in claim 17, wherein said transaction value comprising a unique identifier used to initiate a storage volume ownership request and to track a corresponding storage volume ownership response.

19. The method in claim 17, wherein said current status value comprising a state variable that maintains the current state of the transaction, said current state being selected from among a lock requested state, a lock pending state, a lock granted state, and a lock clearing state.

20. The method in claim 17, wherein said time stamp value comprising a value to mark the time when a transaction started and being used to track how long a lock request has been active but not yet granted.

21. The method in claim 16, wherein said storage volume reservation table is updated when the storage volume ownership transaction establishes a lock.

22. The method in claim 1, wherein said method further comprises the step of sending a storage volume ownership request from one controller to another controller requesting a change in the current storage volume reservation table;

said storage volume ownership request using a lock request data structure to contain information to send to another controller to establish a lock.

23. The method in claim 22, wherein said lock request data structure is a data structure that is sent as a message to another controller in order to establish a lock on a storage volume and includes a parameter identifying the starting sector for this lock region, a parameter identifying the number of sectors to force to disk and invalidate, a parameter that provides a unique identifier to allow other controllers which received the lock requests to determine who made the request, a parameter used to uniquely identify which lock is being serviced, a parameter that identifies the storage volume for which the lock is being requested, and a parameter which identifies the type of lock being requested;

said type of lock being requesting being a reservation, full, partial, or block.

24. The method in claim 23, wherein said primary lock request parameter identifies the region of the storage volume which must be flushed before the host requests can be serviced by the requesting controller, and said secondary lock request parameter identifies the region of the storage volume which needs to be flushed before the lock can be coalesced with another lock or before the entire data extent of the storage volume can be locked.

25. The method of claim 22, further comprising the step of generating a storage volume ownership response message in response to said storage volume ownership request message.

26. The method of claim 25, wherein said storage volume ownership response message includes data fields which pass back of the status of the request and the transaction value.

27. In a computing environment that allows data access through two or more dual-active caching storage-device controllers, a storage device access reservation system comprising:

a storage volume reservation table data structure defined in separate memories within each of said two or more dual-active caching storage-device controllers that stores information indicating which of said controllers has access rights to particular storage volume or logical unit regions;

said storage volume reservation table having a plurality of reservation table entries indexed by storage volume number, each said reservation table entry including an ownership field, a hash table of locks, and a linked list of currently blocked data extents, said reservation table entry applying to all or a portion of a logical unit or storage volume; and

said storage volume reservation table data structure providing consistent information in each of said controllers.

28. A computer program product for maintaining cache coherency amongst a plurality of dual-active caching storage controllers operating in unison supplying data in response to requests from one or more host computers, the computer program product including a computer readable medium and a computer mechanism stored thereon, the computer mechanism performing a method comprising:

defining a storage volume reservation table data structure to maintain reserved, partial, and full ownership status of data extents that are part of the logical unit or storage volume, said ownership status applying to all or a portion of a logical unit or storage volume; and

using said storage volume reservation table data structure to verify that a new update to the data is allowed.

29. The computer program product as in claim 28, wherein said storage volume reservation table data structure having a plurality of reservation table entries indexed by storage volume number, each said reservation table entry including an ownership field, a hash table of locks, and a linked list of currently blocked data extents.

30. A method for accessing data on a storage volume which is accessible by a plurality of multiple-active controllers by a host computer, said method comprising:

receiving a host system read request command or a host system write request command;

implicitly generating a reservation for each said read/write command to ensure that the data accessed by the host is synchronized with data recorded on the storage volume, said implicit reservations being represented by locks in a storage volume reservation table, said reservation applying to all or a portion of said storage volume; and

said storage volume reservation table including a storage volume identifier, a plurality of storage volume state parameters including parameters indicating that the lock is full, partial, or a block and that the reservation is local or remote, and a hash table of locks.

31. The method in claim 30, wherein said storage volume state field within said storage volume reservation entry data structure providing a set of six flags to allow a quick check to be made to determine the state of reservations for a storage volume;

said flags indicating that there are no locks, partial locks, block lock, local reservation, or remote reservation.

32. The method in claim 31, wherein said no lock status indicating that nobody currently owns any locks;

said partial lock status indicating that the controller in which the storage volume reservation table is stored owns pieces of the storage volume, but not the entire storage volume;

said block lock status indicating a temporary blocking lock is in place for this storage volume and usually indicating that another controller is performing an operation;

said local reservation status indicating that this controller has a SCSI Reserve command outstanding for this storage volume, and said remote reservation status indicating that another controller has a SCSI Reserve command outstanding for this storage volume.

33. The method in claim 30, wherein a controller processes a host system read request for the controller to read user data from a specified extent on the storage volume by the steps of:

receiving said host read request command;

directing said host read command to a logical unit number (LUN) and storage volume of the controller;

checking the storage volume reservation table volume state field for the storage volume identified by the LUN associated with the host read request command;

if the controller has a fill ownership the controller proceeds to accept the command and process the host request, if the controller does not have full ownership the controller checks to see if another controller has a device reservation, and if another controller has a device reservation the controller rejects the host request;

said controller checking to determine if said controller owns a partial lock which covers the extent associated with the host read request command by;

(i) examining a partial lock bit of a volume state field to determine if the controller owns any partial locks, and (ii) if the controller owns a partial lock then examining the locks in a hash table of locks to determine if a lock that it owns encloses the extent associated with the host read request command; and

if the controller owns an enclosing lock the controller accepts the command, and if the controller does not own an enclosing lock the controller requests a lock from the other controller and upon receiving an acknowledgment that the lock has been granted the controller accepts the command or upon not receiving an acknowledgment the controller terminates the host read request command.

34. The method in claim 33, wherein said step of accepting said command including steps of:

(i) if said read or write command is accepted the controller determining if the command covers a region which is blocked, and if the determination indicates that the command does not cover a region which is blocked, the command is processed and may move data into a cache; and

(ii) if the determination indicates that the command does cover a region which is blocked, then the controller queues the command to be rescheduled when the block of region has been released, said command being processed by said controller after the lock has been released.

35. The method in claim 30, wherein a controller processes a host system write request for the controller to write user data to a specified extent on the storage volume by the steps of:

a controller processes a host system write request for the controller to write user data to a specified extent on the storage volume by;

receiving said host write request command;

directing said host write command to a logical unit number (LUN) and storage volume of the controller;

checking the storage volume reservation table volume state field for the storage volume identified by the LUN associated with the host write request command;

if the controller has a full ownership the controller proceeds to accept the command and process the host request, if the controller does not have full ownership the controller checks to see if another controller has a device reservation, and if another controller has a device reservation the controller rejects the host request;

said controller checking to determine if said controller owns a partial lock which covers the extent associated with the host write request command by;

(i) examining a partial lock bit of a volume state field to determine if the controller owns any partial locks, and (ii) if the controller owns a partial lock then examining the locks in a hash table of locks to determine if a lock that it owns encloses the extent associated with the host write request command; and

if the controller owns an enclosing lock the controller accepts the command, and if the controller does not own an enclosing lock the controller requests a lock from the other controller and upon receiving an acknowledgment that the lock has been granted the controller accepts the command or upon not receiving an acknowledgment the controller terminates the host write request command.

36. The method in claim 35, wherein said step of accepting said command including steps of:

(i) if said read or write command is accepted the controller determining if the command covers a region which is blocked, and if the determination indicates that the command does not cover a region which is blocked, the command is processed and may move data into a cache; and

(ii) if the determination indicates that the command does cover a region which is blocked, then the controller queues the command to be rescheduled when the block of region has been released, said command being processed by said controller after the lock has been released.

37. In a computing environment that allows data access to a storage volume through two or more active caching controllers, a storage access reservation system comprising:

a storage volume reservation table data structure defined in a separate memory within each of said two or more caching controllers;

said storage volume reservation table having a plurality of reservation table entries indexed by storage volume number, each said reservation table entry including an ownership field, a hash table of locks, and a linked list of currently blocked data extents, said reservation entry field identifying reservation status applying to all or a portion of said storage volume; and

said reservation table data structure providing consistent information in each of said controllers.

38. The storage reservation system in claim 37, wherein said reservation table data structure includes a Logical Unit Number (LUN) reservation table.

39. The storage reservation system in claim 37, wherein said reservation table data structure explicitly reserving storage volumes using any current storage volume (or LUN) reserve commands.

40. The storage reservation system in claim 37, wherein said reservation table data structure implicitly reserving storage volumes using a write operation.

41. The storage reservation system in claim 37, wherein said system providing the capability of invalidating a controller'"'"'s cache based on acquiring new reservation.