Distributed file system and method

• US 8,788,530 B2
• Filed: 10/14/2011
• Issued: 07/22/2014
• Est. Priority Date: 11/01/2002
• Status: Expired due to Fees

• Current Assignee: Hitachi Data Systems Engineering Uk Limited (Hitachi, Ltd.)
• Original Assignee: Hitachi Data Systems Engineering Uk Limited (Hitachi, Ltd.)
• Inventors: Lacapra, Francesco, Cattaneo, Fiorenzo, Benham, Simon L., Willis, Trevor E., Aston, Christopher J.
• Primary Examiner(s): Pulliam, Christyann
• Assistant Examiner(s): OHBA, MELLISSA M
• Application Number:
  US13/273,493
• Publication Number:
  US 20120036161A1
• Time in Patent Office: 1,012 Days
• Field of Search: 707/783
• US Class Current: 707/783
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: Y10S 707/947 Human resources

Abstract

A distributed file system and method distributes file system objects across multiple self-contained volumes, where each volume is owned by a unique file system node. Logical links are used to reference a file system object between volumes. Each file system node includes a relocation directory in which is maintained hard links to locally-stored file system objects that are referenced from another file system node using logical links. Various file system operations that involve multiple volumes are performed without having to place a write lock on more than one volume at a time. Various caching schemes allow the various file system nodes to cache file system object data and metadata.

Claims

1. Apparatus for operating as a cluster node in a file server cluster having a plurality of interconnected cluster nodes, each of which has access to a plurality of logical storage volumes, to provide access by clients, in communication with the nodes, to data on the volumes, the apparatus comprising:

a data storage volume for storing file system objects, each file system object having a unique parent node, the data storage volume deemed to be owned by the apparatus such that only the apparatus is permitted to write to the data storage volume;

at least one directory for storing references to file system objects stored in the various data storage volumes; and

hardware logic comprising object storage logic configured to store file system objects in the data storage volume and to store, in the at least one directory, logical references to file system objects stored in data storage volumes owned by other cluster nodes for which the apparatus is considered to be the parent node and hard references to file system objects stored in the data storage volume for which another node is considered to be the parent node;

each of said hard references including a hard link to a file system object created in a relocation directory of the data storage volume and a unique association between the hard link and a logical identifier used by the parent node to symbolically identify the file system object, wherein the hard link increases a reference count associated with the file system object in the data storage volume independent of the disposition of the logical identifier in the parent node such that the data storage volume is self-contained while allowing the file system object to be accessed symbolically from the parent node using the logical identifier without a physical cross-volume link, and wherein the hard link, the logical identifier, and the unique association between the hard link and the logical identifier represents a logical link between the data storage volume and the parent node that forms;

a physical namespace invisible to the clients of the plurality of nodes, the physical namespace implemented through the plurality of volumes, each volume having an internal hierarchy that connects file system objects through hard links including the hard link in the relocation directory of the data storage volume; and

a logical namespace visible to the clients of the plurality of nodes that spans the entire file system across volumes and is made of file system objects connected via hard links and logical links such that the difference between hard links and logical links is hidden from the clients.

2. The apparatus according to claim 1, wherein:

each of said logical references comprises a logical identifier for accessing the corresponding file system object stored in a data storage volume owned by another node.

3. The apparatus according to claim 1, wherein the object storage logic is configured to store the file system objects in the data storage volume and store the references in the at least one directory while placing a write-lock on no more than one volume at a time, so that multi-volume locking is avoided.

4. The apparatus according to claim 3, wherein the object storage logic is assigned the role of write-lock manager for the various data storage volumes in the course of performing such operations.

5. The apparatus according to claim 2, wherein the relocation directory has a hierarchical structure for storing the hard references.

6. The apparatus according to claim 2, wherein the object storage logic is configured to assign the logical identifier for each hard reference so that each such logical identifier is unique within the data storage volume.

7. The apparatus according to claim 6, wherein the object storage logic is configured to assign a unique identifier based on the count of previously assigned logical identifiers for file system objects in the data storage volume.

8. The apparatus according to claim 1, wherein the object storage logic is configured to permit reads to the data storage volume to be made by any of the nodes.

9. The apparatus according to claim 8, further comprising a node cache and a volume cache associated with the data storage volume, wherein the object storage logic is configured to handle a request to read a sector in a data storage volume owned by another node by

issuing a first read request, for the sector, with a read-only lock to the other node and a second read request, for the sector, to the data storage volume owned by the other node.

10. The apparatus according to claim 8, further comprising a node cache and a volume cache associated with the data storage volume, wherein the object storage logic is configured to handle a read request, for a sector in the data storage volume, with a read-only lock, by

determining whether the sector is stored in the volume cache;

if not, granting the read-only lock and satisfying the request;

if so, and if the sector is not write-locked, granting the read-only lock and sending the sector content from cache; and

if so, and if the sector is write-locked, stalling affirmative action on the request until the read-only lock can be granted and then resuming processing as for when the sector is not write-locked.

11. The apparatus according to claim 10, wherein the object storage logic is configured to record the identity of the other node and of the sector upon granting the read-only lock, so that caching states may be tracked.

12. The apparatus according to claim 2, wherein each logical reference references a corresponding entry in a relocation directory of a data storage volume in another node.

13. The apparatus according to claim 12, wherein each logical reference further comprises:

a numeric value that encodes a numeric string representing a name of the referenced file system object in the relocation directory of the data storage volume in the other node.

14. The apparatus according to claim 1, wherein the object storage logic is configured to maintain for each hard reference a reference back to the corresponding parent node.

15. The apparatus according to claim 14, wherein the reference back to the corresponding parent node comprises:

a volume identifier associated with the data storage volume in the parent node; and

a node identifier associated with the directory containing the logical reference in the data storage volume in the parent node.

16. The apparatus according to claim 14, wherein the reference back to the corresponding parent node comprises:

a variable-length pathname to the directory containing the logical reference in the data storage volume in the parent node.

17. The apparatus according to claim 14, wherein the object storage logic is configured to store the reference back to the corresponding parent node along with the hard reference in the at least one directory.

18. The apparatus according to claim 14, wherein the object storage logic is configured to store the reference back to the corresponding parent node as a separate file system object in the data storage volume.

19. The apparatus according to claim 1, wherein the object storage logic is configured to create a new file system object by

storing a new file system object in the data storage volume;

assigning a logical identifier for a hard reference to the new file system object;

storing the hard reference to the new file system object in the at least one directory; and

providing the logical identifier to another node for creating a logical reference to the new file system object by a data storage volume owned by the other node.

20. The apparatus according to claim 1, wherein the object storage logic is configured to delete a file system object for which the apparatus is considered to be the parent node by

deleting the file system object from the data storage volume;

and

deleting the hard reference to the file system object from the at least one directory.

21. The apparatus according to claim 20, wherein the object storage logic is configured to cause the corresponding parent node to delete the logical reference to the deleted file system object.

22. The apparatus according to claim 1, wherein the object storage logic is configured to delete a file system object for which another node is considered to be the parent node by

deleting the logical reference to the deleted file system object from the at least one directory.

23. The apparatus according to claim 1, wherein the object storage logic is configured to delete the hard reference for a file system object that no longer exists in the data storage volume.

24. The apparatus according to claim 23, wherein the object storage logic comprises a scavenger process for examining the hard references from time to time to determine whether the file system objects referenced by the hard references exist.

25. The apparatus according to claim 1, wherein the object storage logic is configured to delete the logical reference for a file system object that no longer exists in a data storage volume of another node.

26. The apparatus according to claim 25, wherein the object storage logic comprises a scavenger process for examining the logical references from time to time to determine whether the file system objects referenced by the logical references exist.

27. The apparatus according to claim 1, wherein the object storage logic is configured to relocate a file system object stored in the data storage volume to another node by

assigning a logical identifier for a hard reference to the file system object;

storing the hard reference to the file system object in the at least one directory; and

providing the logical identifier to the other node for creating a logical reference to the file system object by a data storage volume owned by the other node.

28. The apparatus according to claim 1, wherein the object storage logic is configured to relocate a file system object stored in a data storage volume of another node by

obtaining a logical identifier from the other node;

and

storing in the at least one directory a logical reference including the logical identifier.

29. The apparatus according to claim 1, further comprising at least one cache for storing a copy of data relating to file system objects stored in other nodes, wherein the object storage logic is configured to invalidate the copy of data relating to a particular file system object upon learning that the file system object was modified by the node in which the file system object is stored.

30. The apparatus according to claim 29, wherein the object storage logic is configured to reacquire the data relating to the file system object.

31. The apparatus according to claim 1, wherein the object storage logic is configured to maintain, for each file system object stored in the data storage volume, a list of other nodes having a copy of data relating to the file system object and to notify each such other node upon modifying a particular file system object.

32. The apparatus according to claim 29, wherein the at least one cache comprises a metadata cache for storing a copy of metadata associated with a file system object, and wherein the object storage logic is configured to operate the metadata cache independently of any other cache used for storing a copy of file system object data.

33. The apparatus according to claim 32, wherein the object storage logic is configured to require another node to obtain a read lock before reading metadata relating to a file system object stored in the data storage volume.

34. The apparatus according to claim 29, wherein the at least one cache comprises a data cache for storing file system object data and a metadata cache for storing metadata, and wherein the object storage logic is configured to map each datum in the data cache to corresponding metadata in the metadata cache.

35. The apparatus according to claim 1, wherein the at least one directory comprises a local undo directory, and wherein the object storage logic is configured to temporarily store in the local undo directory references to file system objects and logical references that have been replaced with a logical reference or another logical reference and to use the local undo directory to recover from a failure during a multi-volume operation that affects the data storage volume.

36. Apparatus for operating as a cluster node in a file server cluster having a plurality of interconnected cluster nodes, each of which has access to a plurality of logical storage volumes, to provide access by clients, in communication with the nodes, to data on the volumes, the apparatus comprising:

a data storage volume for storing file system objects, each file system object having a unique parent node, the data storage volume deemed to be owned by the cluster node such that only the cluster node is permitted to write to the data storage volume;

at least one directory for storing references to file system objects stored in the various data storage volumes;

means for storing file system objects in the data storage volume;

means for storing, in the at least one directory, logical references to file system objects stored in data storage volumes owned by other cluster nodes for which the cluster node is considered to be the parent node; and

means for storing, in the at least one directory, hard references to file system objects stored in the data storage volume for which another cluster node is considered to be the parent node;

each of said hard references including a hard link to a file system object created in a relocation directory of the data storage volume and a unique association between the hard link and a logical identifier used by the parent node to symbolically identify the file system object, wherein the hard link increases a reference count associated with the file system object in the data storage volume independent of the disposition of the logical identifier in the parent node such that the data storage volume is self-contained while allowing the file system object to be accessed symbolically from the parent node using the logical identifier without a physical cross-volume link, and wherein the hard link, the logical identifier, and the unique association between the hard link and the logical identifier represents a logical link between the data storage volume and the parent node that forms;

a physical namespace invisible to the clients of the plurality of nodes, the physical namespace implemented through the plurality of volumes, each volume having an internal hierarchy that connects file system objects through hard links including the hard link in the relocation directory of the data storage volume; and

a logical namespace visible to the clients of the plurality of nodes that spans the entire file system across volumes and is made of file system objects connected via hard links and logical links such that the difference between hard links and logical links is hidden from the clients.

37. Apparatus comprising at least one non-transitory tangible computer readable medium encoded with instructions which, when loaded into a computer, establish processes for operating as a cluster node in a file server cluster having a plurality of interconnected cluster nodes, each of which has access to a plurality of logical storage volumes, to provide access by clients, in communication with the nodes, to data on the volumes, the instructions comprising:

instructions for storing file system objects in a data storage volume, each file system object having a unique parent node, the data storage volume deemed to be owned by the cluster node such that only the cluster node is permitted to write to the data storage volume;

instructions for storing, in at least one directory, logical references to file system objects stored in data storage volumes owned by other cluster nodes for which the cluster node is considered to be the parent node; and

instructions for storing, in the at least one directory, hard references to file system objects stored in the data storage volume for which another cluster node is considered to be the parent node;

each of said hard references including a hard link to a file system object created in a relocation directory of the data storage volume and a unique association between the hard link and a logical identifier used by the parent node to symbolically identify the file system object, wherein the hard link increases a reference count associated with the file system object in the data storage volume independent of the disposition of the logical identifier in the parent node such that the data storage volume is self-contained while allowing the file system object to be accessed symbolically from the parent node using the logical identifier without a physical cross-volume link, and wherein the hard link, the logical identifier, and the unique association between the hard link and the logical identifier represents a logical link between the data storage volume and the parent node that forms;

a physical namespace invisible to the clients of the plurality of nodes, the physical namespace implemented through the plurality of volumes, each volume having an internal hierarchy that connects file system objects through hard links including the hard link in the relocation directory of the data storage volume; and

a logical namespace visible to the clients of the plurality of nodes that spans the entire file system across volumes and is made of file system objects connected via hard links and logical links such that the difference between hard links and logical links is hidden from the clients.