System and method for managing hard and soft lock state information in a distributed storage system environment

• US 7,730,258 B1
• Filed: 11/02/2006
• Issued: 06/01/2010
• Est. Priority Date: 11/01/2005
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Jernigan, Richard P. IV, Noveck, David B., Smith, Toby, Griess, Peter, Hyer, Robert Wyckoff Jr., Kazar, Michael
• Primary Examiner(s): Chace; Christian P
• Assistant Examiner(s): Farrokh; Hashem
• Application Number:
  US11/591,738
• Time in Patent Office: 1,307 Days
• Field of Search: 711/114, 707/8, 714/6
• US Class Current: 711/114
• CPC Class Codes: G06F 12/0806 Multiuser, multiprocessor or multiprocessing cache systems
• CPC Class Codes: G06F 12/0815 Cache consistency protocols
• CPC Class Codes: G06F 12/0866 for peripheral storage systems, e.g. disk cache
• CPC Class Codes: G06F 12/0875 with dedicated cache, e.g. instruction or stack
• CPC Class Codes: G06F 12/14 Protection against unauthorised use of memory or access to memory
• CPC Class Codes: G06F 2212/284 being distributed
• CPC Class Codes: G06F 3/0622 in relation to access
• CPC Class Codes: G06F 3/0637 Permissions
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A system and method manages lock state information in a storage system. A meta-data volume node includes a lock state database which is a comprehensive source for lock state information about data containers in the system. A plurality of data volume nodes include local lock caches, which contain information about locks. Lock state messaging between the meta-data volume node and the data volume nodes is used to assign locks and to update local lock caches. The meta-data volume node is configured to determine whether input/output requests should be allowed or denied and to instruct the data volume nodes of this input/output operation result. Lock information is also sent to the data volume nodes for storage in local lock state caches to the extent the data volume nodes have the network capacity locally.

Claims

1. A method of managing lock state information in a system including a plurality of data volumes of a striped volume set, comprising:

connecting two or more nodes together to form a cluster, t each node in the cluster being associated with least one data volume of the plurality of data volumes and each data volume is a logical arrangement of a plurality of storage devices connected to one of the nodes;

striping one or more data containers across the plurality of volumes in the striped volume set;

configuring a meta-data volume node that includes a lock state database that stores lock state information for one or more data containers on the striped volume set; and

communicating the lock state information including an input/output operation result for input/output requests from said meta-data volume node to one or more data volume nodes of the system, using lock state command messages;

in response to said meta-data volume receiving a client request to add a new lock to a file in a striped volume environment, consulting at said meta-data volume node, a striping table to determine striping rules for said file;

using striping rules from said striping table, and a file range of the new lock being added, and determining at said meta-data volume, which data volume nodes are impacted by the addition of the new lock; and

for each such data volume node listed in a list of impacted data volume nodes, performing at said meta-data volume the following;

if the new lock overlaps with a byte range with no locks present, or locks which are marked as having been cached by the data volume node in question, then the meta-data volume node sends a message to that data volume node, informing it of the new lock and that the file range covered by the new lock is invalid.

2. The method as defined in claim 1 further comprising establishing in each data volume node a local lock cache for storing the lock state information for the data containers served by the data volume node.

3. The method as defined in claim 2 including establishing a set of caching rules for data that is to be stored in said local lock cache depending upon the memory capacity of the local lock cache.

4. The method as defined in claim 1, further comprising:

in response to said meta-data volume node receiving a request from a client for exclusive access to a range of a data container, searching said lock state database for conflicting locks, and if no such locks are found, granting a lock with respect to said range; and

constructing a lock state database structure in said lock state database, said lock state data structure adapted to send messages to a data volume node, the messages including information about the client requesting the exclusive access, a lock state access type, a lock access range and an input/output operation result.

5. The method as defined in claim 4, further comprising:

in response to a receipt from a client that it no longer requires exclusivity, updating the lock state database at the metadata volume node to remove the respective lock; and

inspecting the records at the metadata volume node and sending a lock purge message to one or more data volume nodes that have cached the respective lock; and

updating the records at the metadata volume node to indicate that said data volume nodes have been notified of the removal of the respective lock.

6. The method as defined in claim 5, further comprising:

in response to receipt of a lock purge message, each of said one or more data volume nodes purging cached locks as instructed and sending an acknowledgement to the metadata volume node that such locks have been purged.

7. The method as defined in claim 1, further comprising:

in response to receiving a request for a data access operation, inspecting a local lock cache of a data volume node to determine whether there is a lock that affects the requested operation; and

in the absence of lock state information in the local lock cache, querying the metadata volume node regarding lock states which affect the pending access operation.

8. The method as defined in claim 7, further comprising:

in response to a query from a data volume node, inspecting the lock state database of the metadata volume node for information regarding locks that affect or support the operation; and

sending a message to the data volume node notifying the data volume node of the input/output operation result and locks affecting the pending operation.

9. The method as defined in claim 8, further comprising:

updating records at the metadata volume node to indicate that the recipient data volume node has been notified of a particular lock.

10. The method as defined in claim 9, further comprising:

in response to receiving a message from the metadata volume node, acknowledging the message at the data volume node and handling the access operation according to the input/output operation result provided to it by the meta-data volume node.

11. The method as defined in claim 1 wherein said meta-data volume node is further configured to include a Boolean value indicating whether the data volume node has a complete set of lock state information regarding a file or range.

12. The method as defined in claim 1 including using said lock state command messages in a distributed storage system.

13. The method as defined in claim 1 wherein said meta-data volume node and said one or more data volumes nodes are configured in a distributed storage system.

14. A method of managing lock state information in a system including a plurality of data volumes of a striped volume set, comprising:

configuring a meta-data volume node that includes a lock state database that stores lock state information for one or more data containers on the striped volume set;

communicating the lock state information including an input/output operation result for input/output requests from said meta-data volume node to one or more data volume nodes of the system, using lock state command messages;

in response to said meta-data volume node receiving a request from a client for exclusive access to a range of a data container, searching said lock state database for conflicting locks, and if no such locks are found, granting a lock with respect to said range;

constructing a lock state database structure in said lock state database, said lock state data structure adapted to send messages to a data volume node, the messages including information about the client requesting the exclusive access, a lock state access type, a lock access range and an input/output operation result;

in response to said meta-data volume receiving a client request to add a new lock to a file in a striped volume environment, consulting at said meta-data volume node, a striping table to determine striping rules for said file;

using striping rules from said striping table, and a file range of the new lock being added, and determining at said meta-data volume, which data volume nodes are impacted by the addition of the new lock; and

for each such data volume node listed in a list of impacted data volume nodes, performing at said meta-data volume the following;

if the new lock overlaps with a byte range with no locks present, or locks which are marked as having been cached by the data volume node in question, then the meta-data volume node sends a message to that data volume node, informing it of the new lock and that the file range covered by the new lock is invalid.

15. A system configured to manage lock state information in a system, the system comprising:

two or more nodes connected together to form a cluster, each node in the cluster being associated with at least one data volume of a plurality of data volumes and each data volume is a logical arrangement of a plurality of storage devices connected to the node;

one or more data containers striped across the plurality of volumes in a striped volume set;

a plurality of data volume nodes adapted to receive data access requests from a client directed to a data container or range of the data container; and

a meta-data volume node configured to communicate with said data volume nodes and configured to include a lock state database that stores lock state information for the data container, and to provide input/output operation results to data access requests received from a client at said data volume node;

a striping table to determine striping rules for files, said meta-data volume being configured to consult said striping table to determine striping rules for said file;

said meta-data volume being further configured to use striping rules from said striping table, and a file range of a new lock being added, to determine at said meta-data volume, which data volume nodes are impacted by the addition of the new lock;

said meta-data volume being further configured to receive a message from a data volume regarding client requests for I/O operations, and the meta-data volume checks the lock state database for conflicting locks, and in response to said meta-data volume node finding that there are conflicting locks, then the meta-data volume nodes sends an acknowledgement to the data volume notifying the data volume of the existing conflicting lock and further indicating that the request should be denied;

said meta-data volume node being further configured such that, if there are supporting locks then the meta-date volume sends an access request with supporting lock information and indicating that the request should be allowed;

said meta-data volume node further configured to send an access request acknowledgment message indicating that I/O operation is permitted and a Boolean indication that the data volume issuing the request has all the locks associated with that range after which the data volume node completes the client request.

16. The system as defined in claim 15 further comprising:

at each said data volume node, a local lock cache, the local lock cache for storing lock state information.

17. The system as defined in claim 15, further comprising a storage server including a lock manager for managing lock state information and messaging between said metadata volume node, and one or more data volume nodes.

18. The system as defined in claim 15 wherein said plurality of data volume nodes and said meta-data volume node are configured in a distributed storage system.

19. The system as defined in claim 15 wherein said lock state information relates to a distributed storage system.

20. A computer readable storage medium containing executable program instructions executed by a processor, comprising:

program instructions that connect two or more nodes together to form a cluster, each node in the cluster being associated with at least one data volume of a plurality of data volumes and each data volume is a logical arrangement of a plurality of storage devices connected to the node storing each data volume;

program instructions that stripe one or more data containers across the plurality of volumes in the striped volume set in accordance with a striping table containing striping rules;

program instructions that configure a meta-data volume node that includes a lock state database that stores comprehensive lock state information for one or more data containers stored on the system;

program instructions that communicate lock state information and input/output operation results between said meta-data volume node and one or more data volume nodes of said system, using lock state command messages, said instructions including the meta-data volume node receiving a message from a data volume node regarding client requests for I/O operations,

program instructions that further configure said meta-data volume that in response to receiving a client request for a file in a striped environment, to consult a-at said meta-data volume node, a striping table to determine striping rules for files;

using said striping rules from said striping table, and a file range of a new lock being added, and determining at said meta-data volume, which data volume nodes are impacted by the addition of the new lock;

program instructions that further provide that the meta-data volume checks the lock state database for conflicting locks, and in response to said meta-data volume node finding that there are conflicting locks, then the meta-data volume node sends an acknowledgement to the impacted data volume node notifying it of the existing conflicting locks, and further indicating that the request should be denied;

program instructions that said meta-data volume node further checks the lock state database for supporting locks and if there are supporting locks then the meta-data volume node sends an access request to the impacted data volume node with supporting lock information and indicates that the request should be allowed; and

said meta-data volume node further sends an access request acknowledgment message indicating that I/O operation is permitted and a Boolean indication that the impacted data volume issuing the request has the locks associated with that range after which the data volume node completes the client request.

21. The computer readable storage medium as defined in claim 20 further comprising:

program instructions that establish in each said data volume node, a local lock cache for storing lock state information about data containers served by that data volume node.

22. The computer readable storage medium as defined in claim 20 further comprising:

program instructions that establish a set of caching rules for data that is to be stored in said local lock cache depending upon the memory capacity of the local lock cache.

23. The computer readable storage medium as defined in claim 20 further comprising:

in response to said meta-data volume node receiving a request from a client for exclusive access to a data container or a range of a data container, program instructions that search, by said metadata volume node, its lock state database for conflicting locks, and if no such locks are found, granting the lock with respect to said range; and

program instructions that construct in said lock state database, a lock state data structure for sending messages to an impacted data volume node including information about one or more of the client requesting the lock, a lock state access type, a lock access range, and an input/output operation result.

24. The computer readable storage medium as defined in claim 23 wherein said meta-data volume node further provides a Boolean value indicating whether said impacted data volume node has a substantially complete set of lock state information concerning a file or range.